2024-09-17T22:54:22.387 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-17T22:54:22.410 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-17T22:54:22.501 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-09-17_21:16:02-rbd-squid-distro-default-smithi/7909572 branch: squid description: rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-zstd policy/none rbd-mirror/four-per-cluster supported-random-distro$/{centos_latest} workloads/rbd-mirror-snapshot-stress-workunit-fast-diff} email: ceph-qa@ceph.com first_in_suite: false job_id: '7909572' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi meta: - desc: run two ceph clusters and install rbd-mirror - desc: configure the permissions for client.mirror - desc: 2 ceph clusters with 1 mon and 3 osds each - desc: run four rbd-mirror daemons per cluster - desc: run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon name: teuthology-2024-09-17_21:16:02-rbd-squid-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 3 size: 30 os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: conf: client: debug rbd: 20 debug rbd_mirror: 15 log to stderr: false rbd default features: 125 rbd mirror image policy type: none client.mirror.0: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.1: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.2: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.3: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid global: mon client directed command retry: 5 mon warn on pool no app: false ms inject socket failures: 5000 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bluestore block size: 96636764160 bluestore compression algorithm: zstd bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME sha1: c2ddf9721ef254645201cdbabd9c8f49c30a586f ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: c2ddf9721ef254645201cdbabd9c8f49c30a586f selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: squid sha1: b0d356b98e08f6bc59ef15652f0d0e4298fc83c7 owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - cluster1.mon.a - cluster1.mgr.x - cluster2.mgr.x - cluster1.osd.0 - cluster1.osd.1 - cluster1.osd.2 - cluster1.client.0 - cluster2.client.0 - - cluster2.mon.a - cluster2.osd.0 - cluster2.osd.1 - cluster2.osd.2 - cluster1.client.mirror - cluster1.client.mirror.0 - cluster1.client.mirror.1 - cluster1.client.mirror.2 - cluster1.client.mirror.3 - cluster1.client.mirror.4 - cluster1.client.mirror.5 - cluster1.client.mirror.6 - cluster2.client.mirror - cluster2.client.mirror.0 - cluster2.client.mirror.1 - cluster2.client.mirror.2 - cluster2.client.mirror.3 - cluster2.client.mirror.4 - cluster2.client.mirror.5 - cluster2.client.mirror.6 seed: 1983 sha1: c2ddf9721ef254645201cdbabd9c8f49c30a586f sleep_before_teardown: 0 subset: 79/128 suite: rbd suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_b0d356b98e08f6bc59ef15652f0d0e4298fc83c7/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: b0d356b98e08f6bc59ef15652f0d0e4298fc83c7 targets: smithi006.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBI25UHMjorqH5xXKrb+xjNLrKzHw3AquyMACLWNnQVzXSAKmfZF29gIWsRpW0CsJC8C7QZbDaT9NmJu+Vqwsk7o= smithi038.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE1eTtc2bdOQzb/4YSSSQQ8mKGiFq5T8t3u2PuUJ7oED7RK6GtrnDZsTnuisjLHXzLmeaYHBzMxFHTTfoD/Zn6Y= tasks: - install: extra_packages: - rbd-mirror - ceph: cluster: cluster1 - ceph: cluster: cluster2 - exec: cluster1.client.mirror.0: - sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd' - rbd-mirror: client: cluster1.client.mirror.0 thrash: true - rbd-mirror: client: cluster1.client.mirror.1 thrash: true - rbd-mirror: client: cluster1.client.mirror.2 thrash: true - rbd-mirror: client: cluster1.client.mirror.3 thrash: true - rbd-mirror: client: cluster2.client.mirror.0 thrash: true - rbd-mirror: client: cluster2.client.mirror.1 thrash: true - rbd-mirror: client: cluster2.client.mirror.2 thrash: true - rbd-mirror: client: cluster2.client.mirror.3 thrash: true - rbd-mirror-thrash: cluster: cluster1 - rbd-mirror-thrash: cluster: cluster2 - workunit: clients: cluster1.client.mirror: - rbd/rbd_mirror_stress.sh env: CEPH_ARGS: '' MIRROR_POOL_MODE: image RBD_IMAGE_FEATURES: layering,exclusive-lock,object-map,fast-diff RBD_MIRROR_INSTANCES: '4' RBD_MIRROR_MODE: snapshot RBD_MIRROR_USE_EXISTING_CLUSTER: '1' RBD_MIRROR_USE_RBD_MIRROR: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-17_21:16:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-17T22:54:22.501 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_b0d356b98e08f6bc59ef15652f0d0e4298fc83c7/qa; will attempt to use it 2024-09-17T22:54:22.502 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_b0d356b98e08f6bc59ef15652f0d0e4298fc83c7/qa/tasks 2024-09-17T22:54:22.502 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-17T22:54:22.504 INFO:teuthology.task.internal:Checking packages... 2024-09-17T22:54:22.528 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'c2ddf9721ef254645201cdbabd9c8f49c30a586f' 2024-09-17T22:54:22.528 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-17T22:54:22.528 INFO:teuthology.packaging:ref: None 2024-09-17T22:54:22.528 INFO:teuthology.packaging:tag: None 2024-09-17T22:54:22.529 INFO:teuthology.packaging:branch: squid 2024-09-17T22:54:22.529 INFO:teuthology.packaging:sha1: c2ddf9721ef254645201cdbabd9c8f49c30a586f 2024-09-17T22:54:22.529 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-09-17T22:54:22.741 INFO:teuthology.task.internal:Found packages for ceph version 19.2.0-0 2024-09-17T22:54:22.743 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-17T22:54:22.752 INFO:teuthology.task.internal:no buildpackages task found 2024-09-17T22:54:22.752 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-17T22:54:22.766 INFO:teuthology.task.internal:Saving configuration 2024-09-17T22:54:22.782 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-17T22:54:22.789 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-17T22:54:22.813 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi006.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-17_21:16:02-rbd-squid-distro-default-smithi/7909572', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-17 22:49:36.577645', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBI25UHMjorqH5xXKrb+xjNLrKzHw3AquyMACLWNnQVzXSAKmfZF29gIWsRpW0CsJC8C7QZbDaT9NmJu+Vqwsk7o='} 2024-09-17T22:54:22.838 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi038.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-17_21:16:02-rbd-squid-distro-default-smithi/7909572', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-17 22:49:36.575800', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE1eTtc2bdOQzb/4YSSSQQ8mKGiFq5T8t3u2PuUJ7oED7RK6GtrnDZsTnuisjLHXzLmeaYHBzMxFHTTfoD/Zn6Y='} 2024-09-17T22:54:22.838 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-17T22:54:22.846 INFO:teuthology.task.internal:roles: ubuntu@smithi006.front.sepia.ceph.com - ['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'] 2024-09-17T22:54:22.847 INFO:teuthology.task.internal:roles: ubuntu@smithi038.front.sepia.ceph.com - ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6'] 2024-09-17T22:54:22.847 INFO:teuthology.run_tasks:Running task console_log... 2024-09-17T22:54:22.911 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fc298587d00>, signals=[15]) 2024-09-17T22:54:22.911 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-17T22:54:22.919 INFO:teuthology.task.internal:Opening connections... 2024-09-17T22:54:22.919 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi006.front.sepia.ceph.com 2024-09-17T22:54:22.920 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T22:54:22.998 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi038.front.sepia.ceph.com 2024-09-17T22:54:22.998 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi038.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T22:54:23.074 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-17T22:54:23.081 DEBUG:teuthology.orchestra.run.smithi006:> uname -m 2024-09-17T22:54:23.099 INFO:teuthology.orchestra.run.smithi006.stdout:x86_64 2024-09-17T22:54:23.100 DEBUG:teuthology.orchestra.run.smithi006:> cat /etc/os-release 2024-09-17T22:54:23.157 INFO:teuthology.orchestra.run.smithi006.stdout:NAME="CentOS Stream" 2024-09-17T22:54:23.157 INFO:teuthology.orchestra.run.smithi006.stdout:VERSION="9" 2024-09-17T22:54:23.157 INFO:teuthology.orchestra.run.smithi006.stdout:ID="centos" 2024-09-17T22:54:23.157 INFO:teuthology.orchestra.run.smithi006.stdout:ID_LIKE="rhel fedora" 2024-09-17T22:54:23.158 INFO:teuthology.orchestra.run.smithi006.stdout:VERSION_ID="9" 2024-09-17T22:54:23.158 INFO:teuthology.orchestra.run.smithi006.stdout:PLATFORM_ID="platform:el9" 2024-09-17T22:54:23.158 INFO:teuthology.orchestra.run.smithi006.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-17T22:54:23.158 INFO:teuthology.orchestra.run.smithi006.stdout:ANSI_COLOR="0;31" 2024-09-17T22:54:23.158 INFO:teuthology.orchestra.run.smithi006.stdout:LOGO="fedora-logo-icon" 2024-09-17T22:54:23.158 INFO:teuthology.orchestra.run.smithi006.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-17T22:54:23.158 INFO:teuthology.orchestra.run.smithi006.stdout:HOME_URL="https://centos.org/" 2024-09-17T22:54:23.158 INFO:teuthology.orchestra.run.smithi006.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-17T22:54:23.158 INFO:teuthology.orchestra.run.smithi006.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-17T22:54:23.158 INFO:teuthology.orchestra.run.smithi006.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-17T22:54:23.159 INFO:teuthology.lock.ops:Updating smithi006.front.sepia.ceph.com on lock server 2024-09-17T22:54:23.186 DEBUG:teuthology.orchestra.run.smithi038:> uname -m 2024-09-17T22:54:23.204 INFO:teuthology.orchestra.run.smithi038.stdout:x86_64 2024-09-17T22:54:23.205 DEBUG:teuthology.orchestra.run.smithi038:> cat /etc/os-release 2024-09-17T22:54:23.262 INFO:teuthology.orchestra.run.smithi038.stdout:NAME="CentOS Stream" 2024-09-17T22:54:23.262 INFO:teuthology.orchestra.run.smithi038.stdout:VERSION="9" 2024-09-17T22:54:23.262 INFO:teuthology.orchestra.run.smithi038.stdout:ID="centos" 2024-09-17T22:54:23.262 INFO:teuthology.orchestra.run.smithi038.stdout:ID_LIKE="rhel fedora" 2024-09-17T22:54:23.262 INFO:teuthology.orchestra.run.smithi038.stdout:VERSION_ID="9" 2024-09-17T22:54:23.262 INFO:teuthology.orchestra.run.smithi038.stdout:PLATFORM_ID="platform:el9" 2024-09-17T22:54:23.262 INFO:teuthology.orchestra.run.smithi038.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-17T22:54:23.262 INFO:teuthology.orchestra.run.smithi038.stdout:ANSI_COLOR="0;31" 2024-09-17T22:54:23.262 INFO:teuthology.orchestra.run.smithi038.stdout:LOGO="fedora-logo-icon" 2024-09-17T22:54:23.262 INFO:teuthology.orchestra.run.smithi038.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-17T22:54:23.262 INFO:teuthology.orchestra.run.smithi038.stdout:HOME_URL="https://centos.org/" 2024-09-17T22:54:23.262 INFO:teuthology.orchestra.run.smithi038.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-17T22:54:23.262 INFO:teuthology.orchestra.run.smithi038.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-17T22:54:23.263 INFO:teuthology.orchestra.run.smithi038.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-17T22:54:23.263 INFO:teuthology.lock.ops:Updating smithi038.front.sepia.ceph.com on lock server 2024-09-17T22:54:23.294 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-17T22:54:23.306 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-17T22:54:23.316 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-17T22:54:23.317 DEBUG:teuthology.orchestra.run.smithi006:> test '!' -e /home/ubuntu/cephtest 2024-09-17T22:54:23.319 DEBUG:teuthology.orchestra.run.smithi038:> test '!' -e /home/ubuntu/cephtest 2024-09-17T22:54:23.336 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-17T22:54:23.342 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-17T22:54:23.342 DEBUG:teuthology.orchestra.run.smithi006:> test -z $(ls -A /var/lib/ceph) 2024-09-17T22:54:23.377 DEBUG:teuthology.orchestra.run.smithi038:> test -z $(ls -A /var/lib/ceph) 2024-09-17T22:54:23.404 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-17T22:54:23.467 INFO:teuthology.run_tasks:Running task kernel... 2024-09-17T22:54:23.483 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-17T22:54:23.483 INFO:teuthology.task.kernel:config {'cluster1.mon.a': {'kdb': 1, 'sha1': 'distro'}, 'cluster2.mon.a': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-17T22:54:23.483 DEBUG:teuthology.orchestra.run.smithi006:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-17T22:54:23.484 DEBUG:teuthology.orchestra.run.smithi038:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-17T22:54:23.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T22:54:23.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T22:54:23.501 DEBUG:teuthology.orchestra.run.smithi006:> uname -r 2024-09-17T22:54:23.501 DEBUG:teuthology.orchestra.run.smithi038:> uname -r 2024-09-17T22:54:23.559 INFO:teuthology.orchestra.run.smithi006.stdout:5.14.0-503.el9.x86_64 2024-09-17T22:54:23.559 INFO:teuthology.task.kernel:Running kernel on smithi006: 5.14.0-503.el9.x86_64 2024-09-17T22:54:23.559 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum install -y kernel 2024-09-17T22:54:23.561 INFO:teuthology.orchestra.run.smithi038.stdout:5.14.0-503.el9.x86_64 2024-09-17T22:54:23.561 INFO:teuthology.task.kernel:Running kernel on smithi038: 5.14.0-503.el9.x86_64 2024-09-17T22:54:23.561 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum install -y kernel 2024-09-17T22:54:25.828 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 9 - BaseOS 17 MB/s | 8.3 MB 00:00 2024-09-17T22:54:27.274 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 9 - BaseOS 4.2 MB/s | 8.3 MB 00:01 2024-09-17T22:54:29.092 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 9 - AppStream 29 MB/s | 20 MB 00:00 2024-09-17T22:54:29.179 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 9 - AppStream 9.1 MB/s | 20 MB 00:02 2024-09-17T22:54:33.905 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 9 - CRB 8.5 MB/s | 6.5 MB 00:00 2024-09-17T22:54:34.951 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 9 - CRB 3.4 MB/s | 6.5 MB 00:01 2024-09-17T22:54:35.472 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 9 - Extras packages 63 kB/s | 19 kB 00:00 2024-09-17T22:54:37.210 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 9 - Extras packages 19 kB/s | 19 kB 00:00 2024-09-17T22:54:37.218 INFO:teuthology.orchestra.run.smithi006.stdout:Extra Packages for Enterprise Linux 14 MB/s | 23 MB 00:01 2024-09-17T22:54:38.906 INFO:teuthology.orchestra.run.smithi038.stdout:Extra Packages for Enterprise Linux 14 MB/s | 23 MB 00:01 2024-09-17T22:54:43.065 INFO:teuthology.orchestra.run.smithi006.stdout:lab-extras 44 kB/s | 1.7 kB 00:00 2024-09-17T22:54:44.595 INFO:teuthology.orchestra.run.smithi038.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-09-17T22:54:45.063 INFO:teuthology.orchestra.run.smithi006.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-17T22:54:45.064 INFO:teuthology.orchestra.run.smithi006.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-17T22:54:45.154 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-09-17T22:54:45.159 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-09-17T22:54:45.159 INFO:teuthology.orchestra.run.smithi006.stdout: Package Architecture Version Repository Size 2024-09-17T22:54:45.159 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-09-17T22:54:45.159 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-09-17T22:54:45.159 INFO:teuthology.orchestra.run.smithi006.stdout: kernel x86_64 5.14.0-508.el9 baseos 36 k 2024-09-17T22:54:45.159 INFO:teuthology.orchestra.run.smithi006.stdout:Installing dependencies: 2024-09-17T22:54:45.159 INFO:teuthology.orchestra.run.smithi006.stdout: kernel-core x86_64 5.14.0-508.el9 baseos 16 M 2024-09-17T22:54:45.160 INFO:teuthology.orchestra.run.smithi006.stdout: kernel-modules x86_64 5.14.0-508.el9 baseos 35 M 2024-09-17T22:54:45.160 INFO:teuthology.orchestra.run.smithi006.stdout: kernel-modules-core x86_64 5.14.0-508.el9 baseos 29 M 2024-09-17T22:54:45.160 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T22:54:45.160 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-09-17T22:54:45.160 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-09-17T22:54:45.160 INFO:teuthology.orchestra.run.smithi006.stdout:Install 4 Packages 2024-09-17T22:54:45.160 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T22:54:45.161 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 79 M 2024-09-17T22:54:45.161 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 126 M 2024-09-17T22:54:45.161 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-09-17T22:54:45.400 INFO:teuthology.orchestra.run.smithi006.stdout:(1/4): kernel-5.14.0-508.el9.x86_64.rpm 313 kB/s | 36 kB 00:00 2024-09-17T22:54:46.585 INFO:teuthology.orchestra.run.smithi038.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-17T22:54:46.586 INFO:teuthology.orchestra.run.smithi038.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-17T22:54:46.690 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-09-17T22:54:46.705 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-17T22:54:46.705 INFO:teuthology.orchestra.run.smithi038.stdout: Package Architecture Version Repository Size 2024-09-17T22:54:46.706 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-17T22:54:46.706 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-09-17T22:54:46.706 INFO:teuthology.orchestra.run.smithi038.stdout: kernel x86_64 5.14.0-508.el9 baseos 36 k 2024-09-17T22:54:46.706 INFO:teuthology.orchestra.run.smithi038.stdout:Installing dependencies: 2024-09-17T22:54:46.706 INFO:teuthology.orchestra.run.smithi038.stdout: kernel-core x86_64 5.14.0-508.el9 baseos 16 M 2024-09-17T22:54:46.706 INFO:teuthology.orchestra.run.smithi038.stdout: kernel-modules x86_64 5.14.0-508.el9 baseos 35 M 2024-09-17T22:54:46.706 INFO:teuthology.orchestra.run.smithi038.stdout: kernel-modules-core x86_64 5.14.0-508.el9 baseos 29 M 2024-09-17T22:54:46.706 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T22:54:46.706 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-09-17T22:54:46.706 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-17T22:54:46.706 INFO:teuthology.orchestra.run.smithi038.stdout:Install 4 Packages 2024-09-17T22:54:46.706 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T22:54:46.707 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 79 M 2024-09-17T22:54:46.707 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 126 M 2024-09-17T22:54:46.707 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-09-17T22:54:46.847 INFO:teuthology.orchestra.run.smithi038.stdout:(1/4): kernel-5.14.0-508.el9.x86_64.rpm 302 kB/s | 36 kB 00:00 2024-09-17T22:54:47.447 INFO:teuthology.orchestra.run.smithi038.stdout:(2/4): kernel-core-5.14.0-508.el9.x86_64.rpm 22 MB/s | 16 MB 00:00 2024-09-17T22:54:47.990 INFO:teuthology.orchestra.run.smithi038.stdout:(3/4): kernel-modules-5.14.0-508.el9.x86_64.rpm 27 MB/s | 35 MB 00:01 2024-09-17T22:54:48.067 INFO:teuthology.orchestra.run.smithi006.stdout:(2/4): kernel-core-5.14.0-508.el9.x86_64.rpm 5.6 MB/s | 16 MB 00:02 2024-09-17T22:54:48.631 INFO:teuthology.orchestra.run.smithi038.stdout:(4/4): kernel-modules-core-5.14.0-508.el9.x86_6 16 MB/s | 29 MB 00:01 2024-09-17T22:54:48.632 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-09-17T22:54:48.632 INFO:teuthology.orchestra.run.smithi038.stdout:Total 41 MB/s | 79 MB 00:01 2024-09-17T22:54:49.342 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-09-17T22:54:49.736 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-09-17T22:54:49.736 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-09-17T22:54:50.713 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-09-17T22:54:50.714 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-09-17T22:54:51.533 INFO:teuthology.orchestra.run.smithi006.stdout:(3/4): kernel-modules-5.14.0-508.el9.x86_64.rpm 5.5 MB/s | 35 MB 00:06 2024-09-17T22:54:52.750 INFO:teuthology.orchestra.run.smithi006.stdout:(4/4): kernel-modules-core-5.14.0-508.el9.x86_6 3.9 MB/s | 29 MB 00:07 2024-09-17T22:54:52.750 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-09-17T22:54:52.751 INFO:teuthology.orchestra.run.smithi006.stdout:Total 10 MB/s | 79 MB 00:07 2024-09-17T22:54:53.282 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-09-17T22:54:53.461 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-09-17T22:54:53.849 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-09-17T22:54:53.849 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-09-17T22:54:53.962 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : kernel-modules-core-5.14.0-508.el9.x86_64 1/4 2024-09-17T22:54:54.060 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-17T22:54:54.846 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-09-17T22:54:54.847 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-09-17T22:54:56.819 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-17T22:54:56.946 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-17T22:54:57.411 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-09-17T22:54:58.079 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : kernel-modules-core-5.14.0-508.el9.x86_64 1/4 2024-09-17T22:54:58.159 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-17T22:55:00.941 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-17T22:55:01.081 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-17T22:55:02.064 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-17T22:55:02.105 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : kernel-5.14.0-508.el9.x86_64 4/4 2024-09-17T22:55:06.214 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-17T22:55:06.249 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : kernel-5.14.0-508.el9.x86_64 4/4 2024-09-17T22:55:07.245 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-17T22:55:11.398 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-17T22:55:46.674 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 4/4 2024-09-17T22:55:46.717 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 4/4 2024-09-17T22:55:47.121 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: kernel-5.14.0-508.el9.x86_64 4/4 2024-09-17T22:55:47.121 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : kernel-5.14.0-508.el9.x86_64 1/4 2024-09-17T22:55:47.121 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-17T22:55:47.121 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-17T22:55:47.983 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-17T22:55:47.983 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T22:55:47.983 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-09-17T22:55:47.984 INFO:teuthology.orchestra.run.smithi038.stdout: kernel-5.14.0-508.el9.x86_64 2024-09-17T22:55:47.984 INFO:teuthology.orchestra.run.smithi038.stdout: kernel-core-5.14.0-508.el9.x86_64 2024-09-17T22:55:47.984 INFO:teuthology.orchestra.run.smithi038.stdout: kernel-modules-5.14.0-508.el9.x86_64 2024-09-17T22:55:47.984 INFO:teuthology.orchestra.run.smithi038.stdout: kernel-modules-core-5.14.0-508.el9.x86_64 2024-09-17T22:55:47.984 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T22:55:47.984 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-09-17T22:55:48.351 DEBUG:teuthology.orchestra.run.smithi038:> rpm -q kernel | sort -rV | head -n 1 2024-09-17T22:55:48.441 INFO:teuthology.orchestra.run.smithi038.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-17T22:55:48.441 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-17T22:55:48.441 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-508.el9.x86_64 2024-09-17T22:55:48.442 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-17T22:55:48.442 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-17T22:55:48.442 INFO:teuthology.task.kernel:Installing distro kernel on cluster2.mon.a... 2024-09-17T22:55:48.442 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi038.front.sepia.ceph.com, path=None, version=distro) 2024-09-17T22:55:48.442 DEBUG:teuthology.orchestra.run.smithi038:> rpm -q kernel | sort -rV | head -n 1 2024-09-17T22:55:48.501 INFO:teuthology.orchestra.run.smithi038.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-17T22:55:48.501 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-17T22:55:48.501 DEBUG:teuthology.orchestra.run.smithi038:> sudo rpm -qi grub2-tools 2024-09-17T22:55:48.533 INFO:teuthology.orchestra.run.smithi038.stdout:Name : grub2-tools 2024-09-17T22:55:48.533 INFO:teuthology.orchestra.run.smithi038.stdout:Epoch : 1 2024-09-17T22:55:48.533 INFO:teuthology.orchestra.run.smithi038.stdout:Version : 2.06 2024-09-17T22:55:48.533 INFO:teuthology.orchestra.run.smithi038.stdout:Release : 82.el9 2024-09-17T22:55:48.533 INFO:teuthology.orchestra.run.smithi038.stdout:Architecture: x86_64 2024-09-17T22:55:48.534 INFO:teuthology.orchestra.run.smithi038.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-17T22:55:48.534 INFO:teuthology.orchestra.run.smithi038.stdout:Group : Unspecified 2024-09-17T22:55:48.534 INFO:teuthology.orchestra.run.smithi038.stdout:Size : 8274670 2024-09-17T22:55:48.534 INFO:teuthology.orchestra.run.smithi038.stdout:License : GPLv3+ 2024-09-17T22:55:48.534 INFO:teuthology.orchestra.run.smithi038.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-17T22:55:48.534 INFO:teuthology.orchestra.run.smithi038.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-17T22:55:48.534 INFO:teuthology.orchestra.run.smithi038.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-17T22:55:48.534 INFO:teuthology.orchestra.run.smithi038.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-17T22:55:48.534 INFO:teuthology.orchestra.run.smithi038.stdout:Packager : builder@centos.org 2024-09-17T22:55:48.534 INFO:teuthology.orchestra.run.smithi038.stdout:Vendor : CentOS 2024-09-17T22:55:48.534 INFO:teuthology.orchestra.run.smithi038.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-17T22:55:48.534 INFO:teuthology.orchestra.run.smithi038.stdout:Summary : Support tools for GRUB. 2024-09-17T22:55:48.534 INFO:teuthology.orchestra.run.smithi038.stdout:Description : 2024-09-17T22:55:48.534 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T22:55:48.534 INFO:teuthology.orchestra.run.smithi038.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-17T22:55:48.535 INFO:teuthology.orchestra.run.smithi038.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-17T22:55:48.535 INFO:teuthology.orchestra.run.smithi038.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-17T22:55:48.535 INFO:teuthology.orchestra.run.smithi038.stdout:hardware devices. 2024-09-17T22:55:48.535 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T22:55:48.535 INFO:teuthology.orchestra.run.smithi038.stdout:This subpackage provides tools for support of all platforms. 2024-09-17T22:55:48.536 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-17T22:55:48.536 INFO:teuthology.task.kernel:Updating grub on smithi038 to boot 5.14.0-508.el9.x86_64 2024-09-17T22:55:48.536 DEBUG:teuthology.orchestra.run.smithi038:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-17T22:55:49.198 INFO:teuthology.orchestra.run.smithi038.stderr:Generating grub configuration file ... 2024-09-17T22:55:50.543 INFO:teuthology.orchestra.run.smithi038.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-17T22:55:50.571 INFO:teuthology.orchestra.run.smithi038.stderr:done 2024-09-17T22:55:50.573 DEBUG:teuthology.orchestra.run.smithi038:> mktemp 2024-09-17T22:55:50.588 INFO:teuthology.orchestra.run.smithi038.stdout:/tmp/tmp.dz3gV4MDYm 2024-09-17T22:55:50.589 DEBUG:teuthology.orchestra.run.smithi038:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.dz3gV4MDYm 2024-09-17T22:55:50.655 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod 0666 /tmp/tmp.dz3gV4MDYm 2024-09-17T22:55:50.801 DEBUG:teuthology.orchestra.remote:smithi038:/tmp/tmp.dz3gV4MDYm is 6KB 2024-09-17T22:55:50.850 DEBUG:teuthology.orchestra.run.smithi038:> rm -fr /tmp/tmp.dz3gV4MDYm 2024-09-17T22:55:50.864 DEBUG:teuthology.orchestra.run.smithi038:> sudo /bin/ls /boot/loader/entries || true 2024-09-17T22:55:50.930 INFO:teuthology.orchestra.run.smithi038.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-17T22:55:50.930 INFO:teuthology.orchestra.run.smithi038.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-17T22:55:50.930 INFO:teuthology.orchestra.run.smithi038.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-17T22:55:50.931 INFO:teuthology.orchestra.run.smithi038.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64.conf 2024-09-17T22:55:50.932 DEBUG:teuthology.orchestra.run.smithi038:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64 2024-09-17T22:55:50.996 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 4/4 2024-09-17T22:55:51.038 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 4/4 2024-09-17T22:55:51.091 DEBUG:teuthology.orchestra.run.smithi038:> sudo shutdown -r now 2024-09-17T22:55:51.344 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: kernel-5.14.0-508.el9.x86_64 4/4 2024-09-17T22:55:51.344 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : kernel-5.14.0-508.el9.x86_64 1/4 2024-09-17T22:55:51.345 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-17T22:55:51.345 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-17T22:55:52.181 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-17T22:55:52.181 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T22:55:52.181 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-09-17T22:55:52.181 INFO:teuthology.orchestra.run.smithi006.stdout: kernel-5.14.0-508.el9.x86_64 2024-09-17T22:55:52.181 INFO:teuthology.orchestra.run.smithi006.stdout: kernel-core-5.14.0-508.el9.x86_64 2024-09-17T22:55:52.181 INFO:teuthology.orchestra.run.smithi006.stdout: kernel-modules-5.14.0-508.el9.x86_64 2024-09-17T22:55:52.182 INFO:teuthology.orchestra.run.smithi006.stdout: kernel-modules-core-5.14.0-508.el9.x86_64 2024-09-17T22:55:52.182 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T22:55:52.182 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-09-17T22:55:52.517 DEBUG:teuthology.orchestra.run.smithi006:> rpm -q kernel | sort -rV | head -n 1 2024-09-17T22:55:52.610 INFO:teuthology.orchestra.run.smithi006.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-17T22:55:52.611 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-17T22:55:52.611 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-508.el9.x86_64 2024-09-17T22:55:52.611 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-17T22:55:52.611 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-17T22:55:52.611 INFO:teuthology.task.kernel:Installing distro kernel on cluster1.mon.a... 2024-09-17T22:55:52.612 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi006.front.sepia.ceph.com, path=None, version=distro) 2024-09-17T22:55:52.612 DEBUG:teuthology.orchestra.run.smithi006:> rpm -q kernel | sort -rV | head -n 1 2024-09-17T22:55:52.674 INFO:teuthology.orchestra.run.smithi006.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-17T22:55:52.674 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-17T22:55:52.674 DEBUG:teuthology.orchestra.run.smithi006:> sudo rpm -qi grub2-tools 2024-09-17T22:55:52.705 INFO:teuthology.orchestra.run.smithi006.stdout:Name : grub2-tools 2024-09-17T22:55:52.705 INFO:teuthology.orchestra.run.smithi006.stdout:Epoch : 1 2024-09-17T22:55:52.705 INFO:teuthology.orchestra.run.smithi006.stdout:Version : 2.06 2024-09-17T22:55:52.705 INFO:teuthology.orchestra.run.smithi006.stdout:Release : 82.el9 2024-09-17T22:55:52.705 INFO:teuthology.orchestra.run.smithi006.stdout:Architecture: x86_64 2024-09-17T22:55:52.706 INFO:teuthology.orchestra.run.smithi006.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-17T22:55:52.706 INFO:teuthology.orchestra.run.smithi006.stdout:Group : Unspecified 2024-09-17T22:55:52.706 INFO:teuthology.orchestra.run.smithi006.stdout:Size : 8274670 2024-09-17T22:55:52.706 INFO:teuthology.orchestra.run.smithi006.stdout:License : GPLv3+ 2024-09-17T22:55:52.706 INFO:teuthology.orchestra.run.smithi006.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-17T22:55:52.706 INFO:teuthology.orchestra.run.smithi006.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-17T22:55:52.706 INFO:teuthology.orchestra.run.smithi006.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-17T22:55:52.706 INFO:teuthology.orchestra.run.smithi006.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-17T22:55:52.706 INFO:teuthology.orchestra.run.smithi006.stdout:Packager : builder@centos.org 2024-09-17T22:55:52.706 INFO:teuthology.orchestra.run.smithi006.stdout:Vendor : CentOS 2024-09-17T22:55:52.706 INFO:teuthology.orchestra.run.smithi006.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-17T22:55:52.706 INFO:teuthology.orchestra.run.smithi006.stdout:Summary : Support tools for GRUB. 2024-09-17T22:55:52.706 INFO:teuthology.orchestra.run.smithi006.stdout:Description : 2024-09-17T22:55:52.707 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T22:55:52.707 INFO:teuthology.orchestra.run.smithi006.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-17T22:55:52.707 INFO:teuthology.orchestra.run.smithi006.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-17T22:55:52.707 INFO:teuthology.orchestra.run.smithi006.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-17T22:55:52.707 INFO:teuthology.orchestra.run.smithi006.stdout:hardware devices. 2024-09-17T22:55:52.707 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T22:55:52.707 INFO:teuthology.orchestra.run.smithi006.stdout:This subpackage provides tools for support of all platforms. 2024-09-17T22:55:52.708 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-17T22:55:52.708 INFO:teuthology.task.kernel:Updating grub on smithi006 to boot 5.14.0-508.el9.x86_64 2024-09-17T22:55:52.708 DEBUG:teuthology.orchestra.run.smithi006:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-17T22:55:53.354 INFO:teuthology.orchestra.run.smithi006.stderr:Generating grub configuration file ... 2024-09-17T22:55:54.821 INFO:teuthology.orchestra.run.smithi006.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-17T22:55:54.849 INFO:teuthology.orchestra.run.smithi006.stderr:done 2024-09-17T22:55:54.856 DEBUG:teuthology.orchestra.run.smithi006:> mktemp 2024-09-17T22:55:54.871 INFO:teuthology.orchestra.run.smithi006.stdout:/tmp/tmp.juHivoIQ7I 2024-09-17T22:55:54.871 DEBUG:teuthology.orchestra.run.smithi006:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.juHivoIQ7I 2024-09-17T22:55:54.938 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 0666 /tmp/tmp.juHivoIQ7I 2024-09-17T22:55:55.074 DEBUG:teuthology.orchestra.remote:smithi006:/tmp/tmp.juHivoIQ7I is 6KB 2024-09-17T22:55:55.122 DEBUG:teuthology.orchestra.run.smithi006:> rm -fr /tmp/tmp.juHivoIQ7I 2024-09-17T22:55:55.137 DEBUG:teuthology.orchestra.run.smithi006:> sudo /bin/ls /boot/loader/entries || true 2024-09-17T22:55:55.208 INFO:teuthology.orchestra.run.smithi006.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-17T22:55:55.209 INFO:teuthology.orchestra.run.smithi006.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-17T22:55:55.209 INFO:teuthology.orchestra.run.smithi006.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-17T22:55:55.209 INFO:teuthology.orchestra.run.smithi006.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64.conf 2024-09-17T22:55:55.210 DEBUG:teuthology.orchestra.run.smithi006:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64 2024-09-17T22:55:55.359 DEBUG:teuthology.orchestra.run.smithi006:> sudo shutdown -r now 2024-09-17T22:56:21.119 INFO:teuthology.task.kernel:Checking client cluster2.mon.a for new kernel version... 2024-09-17T22:56:21.120 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi038.front.sepia.ceph.com' 2024-09-17T22:56:21.121 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi038.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T22:56:25.367 INFO:teuthology.task.kernel:Checking client cluster1.mon.a for new kernel version... 2024-09-17T22:56:25.367 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi006.front.sepia.ceph.com' 2024-09-17T22:56:25.368 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T22:57:21.178 DEBUG:teuthology.orchestra.remote:timed out 2024-09-17T22:57:25.373 DEBUG:teuthology.orchestra.remote:timed out 2024-09-17T22:57:30.181 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi038.front.sepia.ceph.com' 2024-09-17T22:57:30.182 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi038.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T22:57:30.639 DEBUG:teuthology.orchestra.run.smithi038:> true 2024-09-17T22:57:31.123 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi038.front.sepia.ceph.com' 2024-09-17T22:57:31.123 INFO:teuthology.task.kernel:Checking kernel version of cluster2.mon.a, want "5.14.0-508.el9.x86_64"... 2024-09-17T22:57:31.123 DEBUG:teuthology.orchestra.run.smithi038:> uname -r 2024-09-17T22:57:31.139 INFO:teuthology.orchestra.run.smithi038.stdout:5.14.0-508.el9.x86_64 2024-09-17T22:57:31.139 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-508.el9.x86_64 vs 5.14.0-508.el9.x86_64 2024-09-17T22:57:31.139 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-17T22:57:31.139 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-17T22:57:32.140 INFO:teuthology.task.kernel:Enabling kdb on cluster2.mon.a... 2024-09-17T22:57:32.140 DEBUG:teuthology.orchestra.run.smithi038:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-17T22:57:32.234 INFO:teuthology.orchestra.run.smithi038.stdout:ttyS1 2024-09-17T22:57:32.260 DEBUG:teuthology.parallel:result is None 2024-09-17T22:57:34.375 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi006.front.sepia.ceph.com' 2024-09-17T22:57:34.376 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T22:57:35.116 DEBUG:teuthology.orchestra.run.smithi006:> true 2024-09-17T22:57:35.570 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi006.front.sepia.ceph.com' 2024-09-17T22:57:35.570 INFO:teuthology.task.kernel:Checking kernel version of cluster1.mon.a, want "5.14.0-508.el9.x86_64"... 2024-09-17T22:57:35.571 DEBUG:teuthology.orchestra.run.smithi006:> uname -r 2024-09-17T22:57:35.587 INFO:teuthology.orchestra.run.smithi006.stdout:5.14.0-508.el9.x86_64 2024-09-17T22:57:35.587 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-508.el9.x86_64 vs 5.14.0-508.el9.x86_64 2024-09-17T22:57:35.587 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-17T22:57:35.587 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-17T22:57:36.589 INFO:teuthology.task.kernel:Enabling kdb on cluster1.mon.a... 2024-09-17T22:57:36.592 DEBUG:teuthology.orchestra.run.smithi006:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-17T22:57:36.671 INFO:teuthology.orchestra.run.smithi006.stdout:ttyS1 2024-09-17T22:57:36.705 DEBUG:teuthology.parallel:result is None 2024-09-17T22:57:36.705 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-17T22:57:36.715 INFO:teuthology.task.internal:Creating test directory... 2024-09-17T22:57:36.716 DEBUG:teuthology.orchestra.run.smithi006:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-17T22:57:36.718 DEBUG:teuthology.orchestra.run.smithi038:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-17T22:57:36.735 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-17T22:57:36.746 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-17T22:57:36.760 INFO:teuthology.task.internal:Creating archive directory... 2024-09-17T22:57:36.761 DEBUG:teuthology.orchestra.run.smithi006:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-17T22:57:36.774 DEBUG:teuthology.orchestra.run.smithi038:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-17T22:57:36.817 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-17T22:57:36.828 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-17T22:57:36.829 DEBUG:teuthology.orchestra.run.smithi006:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-17T22:57:36.833 DEBUG:teuthology.orchestra.run.smithi038:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-17T22:57:36.859 INFO:teuthology.orchestra.run.smithi006.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T22:57:36.870 INFO:teuthology.orchestra.run.smithi006.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T22:57:36.881 INFO:teuthology.orchestra.run.smithi038.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T22:57:36.892 INFO:teuthology.orchestra.run.smithi038.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T22:57:36.893 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-17T22:57:36.905 INFO:teuthology.task.internal:Configuring sudo... 2024-09-17T22:57:36.905 DEBUG:teuthology.orchestra.run.smithi006:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-17T22:57:36.914 DEBUG:teuthology.orchestra.run.smithi038:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-17T22:57:36.961 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-17T22:57:36.975 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-17T22:57:36.975 DEBUG:teuthology.orchestra.run.smithi006:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-17T22:57:36.981 DEBUG:teuthology.orchestra.run.smithi038:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-17T22:57:37.017 DEBUG:teuthology.orchestra.run.smithi006:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-17T22:57:37.082 DEBUG:teuthology.orchestra.run.smithi006:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-17T22:57:37.131 DEBUG:teuthology.orchestra.run.smithi006:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-17T22:57:37.216 DEBUG:teuthology.orchestra.run.smithi006:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-17T22:57:37.256 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-09-17T22:57:37.256 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-17T22:57:37.324 DEBUG:teuthology.orchestra.run.smithi038:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-17T22:57:37.394 DEBUG:teuthology.orchestra.run.smithi038:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-17T22:57:37.467 DEBUG:teuthology.orchestra.run.smithi038:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-17T22:57:37.535 DEBUG:teuthology.orchestra.run.smithi038:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-17T22:57:37.603 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-09-17T22:57:37.603 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-17T22:57:37.668 DEBUG:teuthology.orchestra.run.smithi006:> sudo service rsyslog restart 2024-09-17T22:57:37.671 DEBUG:teuthology.orchestra.run.smithi038:> sudo service rsyslog restart 2024-09-17T22:57:37.720 INFO:teuthology.orchestra.run.smithi006.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-17T22:57:37.805 INFO:teuthology.orchestra.run.smithi038.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-17T22:57:38.304 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-17T22:57:38.314 INFO:teuthology.task.internal:Starting timer... 2024-09-17T22:57:38.314 INFO:teuthology.run_tasks:Running task pcp... 2024-09-17T22:57:38.332 INFO:teuthology.run_tasks:Running task selinux... 2024-09-17T22:57:38.346 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-09-17T22:57:38.346 DEBUG:teuthology.orchestra.run.smithi006:> sudo service auditd rotate 2024-09-17T22:57:38.423 INFO:teuthology.orchestra.run.smithi006.stdout:Rotating logs: 2024-09-17T22:57:38.425 DEBUG:teuthology.orchestra.run.smithi038:> sudo service auditd rotate 2024-09-17T22:57:38.508 INFO:teuthology.orchestra.run.smithi038.stdout:Rotating logs: 2024-09-17T22:57:38.510 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-17T22:57:38.510 DEBUG:teuthology.orchestra.run.smithi006:> /usr/sbin/getenforce 2024-09-17T22:57:38.543 INFO:teuthology.orchestra.run.smithi006.stdout:Permissive 2024-09-17T22:57:38.544 DEBUG:teuthology.orchestra.run.smithi038:> /usr/sbin/getenforce 2024-09-17T22:57:38.667 INFO:teuthology.orchestra.run.smithi038.stdout:Permissive 2024-09-17T22:57:38.667 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi006.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi038.front.sepia.ceph.com': 'permissive'} 2024-09-17T22:57:38.667 DEBUG:teuthology.orchestra.run.smithi006:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-17T22:57:38.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T22:57:38.695 DEBUG:teuthology.orchestra.run.smithi038:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-17T22:57:38.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T22:57:38.723 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-17T22:57:38.723 DEBUG:teuthology.orchestra.run.smithi006:> sudo /usr/sbin/setenforce permissive 2024-09-17T22:57:38.799 DEBUG:teuthology.orchestra.run.smithi038:> sudo /usr/sbin/setenforce permissive 2024-09-17T22:57:38.839 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-17T22:57:38.852 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-17T22:57:38.864 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-17T22:57:38.865 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi006.front.sepia.ceph.com,smithi038.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-17T23:01:31.282 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi006.front.sepia.ceph.com'), Remote(name='ubuntu@smithi038.front.sepia.ceph.com')] 2024-09-17T23:01:31.284 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi006.front.sepia.ceph.com' 2024-09-17T23:01:31.285 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T23:01:31.366 DEBUG:teuthology.orchestra.run.smithi006:> true 2024-09-17T23:01:31.449 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi006.front.sepia.ceph.com' 2024-09-17T23:01:31.449 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi038.front.sepia.ceph.com' 2024-09-17T23:01:31.450 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi038.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T23:01:31.528 DEBUG:teuthology.orchestra.run.smithi038:> true 2024-09-17T23:01:31.604 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi038.front.sepia.ceph.com' 2024-09-17T23:01:31.604 INFO:teuthology.run_tasks:Running task clock... 2024-09-17T23:01:31.615 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-17T23:01:31.615 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-17T23:01:31.616 DEBUG:teuthology.orchestra.run.smithi006:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-17T23:01:31.618 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-17T23:01:31.619 DEBUG:teuthology.orchestra.run.smithi038:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-17T23:01:31.647 INFO:teuthology.orchestra.run.smithi006.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-17T23:01:31.663 INFO:teuthology.orchestra.run.smithi006.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-17T23:01:31.675 INFO:teuthology.orchestra.run.smithi038.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-17T23:01:31.688 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: ntpd: command not found 2024-09-17T23:01:31.689 INFO:teuthology.orchestra.run.smithi038.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-17T23:01:31.700 INFO:teuthology.orchestra.run.smithi006.stdout:506 Cannot talk to daemon 2024-09-17T23:01:31.716 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: ntpd: command not found 2024-09-17T23:01:31.716 INFO:teuthology.orchestra.run.smithi006.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-17T23:01:31.727 INFO:teuthology.orchestra.run.smithi038.stdout:506 Cannot talk to daemon 2024-09-17T23:01:31.730 INFO:teuthology.orchestra.run.smithi006.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-17T23:01:31.741 INFO:teuthology.orchestra.run.smithi038.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-17T23:01:31.756 INFO:teuthology.orchestra.run.smithi038.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-17T23:01:31.795 INFO:teuthology.orchestra.run.smithi006.stderr:bash: line 1: ntpq: command not found 2024-09-17T23:01:31.799 INFO:teuthology.orchestra.run.smithi006.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-17T23:01:31.799 INFO:teuthology.orchestra.run.smithi006.stdout:=============================================================================== 2024-09-17T23:01:31.800 INFO:teuthology.orchestra.run.smithi006.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T23:01:31.800 INFO:teuthology.orchestra.run.smithi006.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T23:01:31.800 INFO:teuthology.orchestra.run.smithi006.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T23:01:31.800 INFO:teuthology.orchestra.run.smithi006.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T23:01:31.812 INFO:teuthology.orchestra.run.smithi038.stderr:bash: line 1: ntpq: command not found 2024-09-17T23:01:31.817 INFO:teuthology.orchestra.run.smithi038.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-17T23:01:31.817 INFO:teuthology.orchestra.run.smithi038.stdout:=============================================================================== 2024-09-17T23:01:31.817 INFO:teuthology.orchestra.run.smithi038.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T23:01:31.817 INFO:teuthology.orchestra.run.smithi038.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T23:01:31.817 INFO:teuthology.orchestra.run.smithi038.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T23:01:31.817 INFO:teuthology.orchestra.run.smithi038.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T23:01:31.818 INFO:teuthology.run_tasks:Running task install... 2024-09-17T23:01:31.826 DEBUG:teuthology.task.install:project ceph 2024-09-17T23:01:31.826 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'c2ddf9721ef254645201cdbabd9c8f49c30a586f'}} 2024-09-17T23:01:31.826 DEBUG:teuthology.task.install:config {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': 'c2ddf9721ef254645201cdbabd9c8f49c30a586f'} 2024-09-17T23:01:31.826 INFO:teuthology.task.install:Using flavor: default 2024-09-17T23:01:31.833 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-17T23:01:31.833 INFO:teuthology.task.install:extra packages: ['rbd-mirror'] 2024-09-17T23:01:31.834 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': ['rbd-mirror'], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'c2ddf9721ef254645201cdbabd9c8f49c30a586f', 'tag': None, 'wait_for_package': False} 2024-09-17T23:01:31.834 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=c2ddf9721ef254645201cdbabd9c8f49c30a586f 2024-09-17T23:01:31.837 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': ['rbd-mirror'], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'c2ddf9721ef254645201cdbabd9c8f49c30a586f', 'tag': None, 'wait_for_package': False} 2024-09-17T23:01:31.837 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=c2ddf9721ef254645201cdbabd9c8f49c30a586f 2024-09-17T23:01:31.984 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/squid/c2ddf9721ef254645201cdbabd9c8f49c30a586f/centos/9/flavors/default/ 2024-09-17T23:01:31.984 INFO:teuthology.task.install.rpm:Package version is 19.1.1-330.gc2ddf972 2024-09-17T23:01:31.987 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/squid/c2ddf9721ef254645201cdbabd9c8f49c30a586f/centos/9/flavors/default/ 2024-09-17T23:01:31.988 INFO:teuthology.task.install.rpm:Package version is 19.1.1-330.gc2ddf972 2024-09-17T23:01:32.095 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/squid/c2ddf9721ef254645201cdbabd9c8f49c30a586f/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/squid/c2ddf9721ef254645201cdbabd9c8f49c30a586f/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/squid/c2ddf9721ef254645201cdbabd9c8f49c30a586f/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-17T23:01:32.095 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-09-17T23:01:32.095 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-17T23:01:32.101 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/squid/c2ddf9721ef254645201cdbabd9c8f49c30a586f/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/squid/c2ddf9721ef254645201cdbabd9c8f49c30a586f/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/squid/c2ddf9721ef254645201cdbabd9c8f49c30a586f/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-17T23:01:32.101 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-09-17T23:01:32.101 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-17T23:01:32.123 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, rbd-mirror on remote rpm x86_64 2024-09-17T23:01:32.124 DEBUG:teuthology.orchestra.run.smithi038:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/c2ddf9721ef254645201cdbabd9c8f49c30a586f/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-17T23:01:32.129 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, rbd-mirror on remote rpm x86_64 2024-09-17T23:01:32.129 DEBUG:teuthology.orchestra.run.smithi006:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/c2ddf9721ef254645201cdbabd9c8f49c30a586f/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-17T23:01:32.191 DEBUG:teuthology.orchestra.run.smithi038:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-17T23:01:32.196 DEBUG:teuthology.orchestra.run.smithi006:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-17T23:01:32.268 DEBUG:teuthology.orchestra.run.smithi038:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-17T23:01:32.273 DEBUG:teuthology.orchestra.run.smithi006:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-17T23:01:32.333 INFO:teuthology.orchestra.run.smithi038.stdout:check_obsoletes = 1 2024-09-17T23:01:32.335 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum clean all 2024-09-17T23:01:32.337 INFO:teuthology.orchestra.run.smithi006.stdout:check_obsoletes = 1 2024-09-17T23:01:32.339 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum clean all 2024-09-17T23:01:32.579 INFO:teuthology.orchestra.run.smithi006.stdout:45 files removed 2024-09-17T23:01:32.581 INFO:teuthology.orchestra.run.smithi038.stdout:45 files removed 2024-09-17T23:01:32.606 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install ceph-radosgw 2024-09-17T23:01:32.608 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-radosgw 2024-09-17T23:01:33.185 INFO:teuthology.orchestra.run.smithi038.stdout:ceph packages for x86_64 301 kB/s | 84 kB 00:00 2024-09-17T23:01:33.195 INFO:teuthology.orchestra.run.smithi006.stdout:ceph packages for x86_64 287 kB/s | 84 kB 00:00 2024-09-17T23:01:33.478 INFO:teuthology.orchestra.run.smithi038.stdout:ceph noarch packages 48 kB/s | 12 kB 00:00 2024-09-17T23:01:33.487 INFO:teuthology.orchestra.run.smithi006.stdout:ceph noarch packages 48 kB/s | 12 kB 00:00 2024-09-17T23:01:33.763 INFO:teuthology.orchestra.run.smithi006.stdout:ceph source packages 8.0 kB/s | 1.9 kB 00:00 2024-09-17T23:01:33.769 INFO:teuthology.orchestra.run.smithi038.stdout:ceph source packages 7.4 kB/s | 1.9 kB 00:00 2024-09-17T23:01:34.316 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 9 - BaseOS 16 MB/s | 8.3 MB 00:00 2024-09-17T23:01:43.169 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 9 - BaseOS 903 kB/s | 8.3 MB 00:09 2024-09-17T23:01:43.341 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 9 - AppStream 2.6 MB/s | 20 MB 00:07 2024-09-17T23:01:45.037 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 9 - AppStream 26 MB/s | 20 MB 00:00 2024-09-17T23:01:48.850 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 9 - CRB 4.2 MB/s | 6.5 MB 00:01 2024-09-17T23:01:50.417 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 9 - Extras packages 69 kB/s | 19 kB 00:00 2024-09-17T23:01:51.371 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 9 - CRB 2.8 MB/s | 6.5 MB 00:02 2024-09-17T23:01:51.980 INFO:teuthology.orchestra.run.smithi006.stdout:Extra Packages for Enterprise Linux 16 MB/s | 23 MB 00:01 2024-09-17T23:01:53.064 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 9 - Extras packages 47 kB/s | 19 kB 00:00 2024-09-17T23:01:54.484 INFO:teuthology.orchestra.run.smithi038.stdout:Extra Packages for Enterprise Linux 17 MB/s | 23 MB 00:01 2024-09-17T23:01:57.636 INFO:teuthology.orchestra.run.smithi006.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-09-17T23:01:59.948 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-09-17T23:01:59.965 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-09-17T23:01:59.965 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repository Size 2024-09-17T23:01:59.965 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-09-17T23:01:59.965 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-09-17T23:01:59.965 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-radosgw x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 11 M 2024-09-17T23:01:59.965 INFO:teuthology.orchestra.run.smithi006.stdout:Upgrading: 2024-09-17T23:01:59.966 INFO:teuthology.orchestra.run.smithi006.stdout: librados2 x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 3.4 M 2024-09-17T23:01:59.966 INFO:teuthology.orchestra.run.smithi006.stdout: librbd1 x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 3.2 M 2024-09-17T23:01:59.966 INFO:teuthology.orchestra.run.smithi006.stdout:Installing dependencies: 2024-09-17T23:01:59.966 INFO:teuthology.orchestra.run.smithi006.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-17T23:01:59.966 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-base x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 5.4 M 2024-09-17T23:01:59.966 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-common x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 21 M 2024-09-17T23:01:59.966 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-selinux x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 25 k 2024-09-17T23:01:59.966 INFO:teuthology.orchestra.run.smithi006.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-17T23:01:59.966 INFO:teuthology.orchestra.run.smithi006.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-17T23:01:59.966 INFO:teuthology.orchestra.run.smithi006.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-17T23:01:59.966 INFO:teuthology.orchestra.run.smithi006.stdout: libcephfs2 x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 751 k 2024-09-17T23:01:59.966 INFO:teuthology.orchestra.run.smithi006.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-17T23:01:59.966 INFO:teuthology.orchestra.run.smithi006.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-17T23:01:59.967 INFO:teuthology.orchestra.run.smithi006.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-17T23:01:59.967 INFO:teuthology.orchestra.run.smithi006.stdout: libradosstriper1 x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 506 k 2024-09-17T23:01:59.967 INFO:teuthology.orchestra.run.smithi006.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-17T23:01:59.967 INFO:teuthology.orchestra.run.smithi006.stdout: librgw2 x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 5.3 M 2024-09-17T23:01:59.967 INFO:teuthology.orchestra.run.smithi006.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-17T23:01:59.967 INFO:teuthology.orchestra.run.smithi006.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-17T23:01:59.967 INFO:teuthology.orchestra.run.smithi006.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-17T23:01:59.967 INFO:teuthology.orchestra.run.smithi006.stdout: python3-ceph-argparse x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 45 k 2024-09-17T23:01:59.967 INFO:teuthology.orchestra.run.smithi006.stdout: python3-ceph-common x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 136 k 2024-09-17T23:01:59.967 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cephfs x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 162 k 2024-09-17T23:01:59.967 INFO:teuthology.orchestra.run.smithi006.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-17T23:01:59.967 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rados x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 320 k 2024-09-17T23:01:59.967 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rbd x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 300 k 2024-09-17T23:01:59.968 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rgw x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 99 k 2024-09-17T23:01:59.968 INFO:teuthology.orchestra.run.smithi006.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-17T23:01:59.968 INFO:teuthology.orchestra.run.smithi006.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-17T23:01:59.968 INFO:teuthology.orchestra.run.smithi006.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-17T23:01:59.968 INFO:teuthology.orchestra.run.smithi006.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-17T23:01:59.968 INFO:teuthology.orchestra.run.smithi006.stdout:Installing weak dependencies: 2024-09-17T23:01:59.968 INFO:teuthology.orchestra.run.smithi006.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-17T23:01:59.968 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:01:59.968 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-09-17T23:01:59.968 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-09-17T23:01:59.968 INFO:teuthology.orchestra.run.smithi006.stdout:Install 30 Packages 2024-09-17T23:01:59.968 INFO:teuthology.orchestra.run.smithi006.stdout:Upgrade 2 Packages 2024-09-17T23:01:59.968 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:01:59.969 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 60 M 2024-09-17T23:01:59.969 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-09-17T23:02:00.182 INFO:teuthology.orchestra.run.smithi038.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-09-17T23:02:01.095 INFO:teuthology.orchestra.run.smithi006.stdout:(1/32): ceph-base-19.1.1-330.gc2ddf972.el9.x86_ 5.1 MB/s | 5.4 MB 00:01 2024-09-17T23:02:01.171 INFO:teuthology.orchestra.run.smithi006.stdout:(2/32): ceph-selinux-19.1.1-330.gc2ddf972.el9.x 336 kB/s | 25 kB 00:00 2024-09-17T23:02:01.321 INFO:teuthology.orchestra.run.smithi006.stdout:(3/32): libcephfs2-19.1.1-330.gc2ddf972.el9.x86 4.9 MB/s | 751 kB 00:00 2024-09-17T23:02:01.422 INFO:teuthology.orchestra.run.smithi006.stdout:(4/32): libradosstriper1-19.1.1-330.gc2ddf972.e 4.9 MB/s | 506 kB 00:00 2024-09-17T23:02:01.838 INFO:teuthology.orchestra.run.smithi006.stdout:(5/32): ceph-radosgw-19.1.1-330.gc2ddf972.el9.x 5.9 MB/s | 11 MB 00:01 2024-09-17T23:02:01.922 INFO:teuthology.orchestra.run.smithi006.stdout:(6/32): python3-ceph-argparse-19.1.1-330.gc2ddf 543 kB/s | 45 kB 00:00 2024-09-17T23:02:02.014 INFO:teuthology.orchestra.run.smithi006.stdout:(7/32): python3-ceph-common-19.1.1-330.gc2ddf97 1.5 MB/s | 136 kB 00:00 2024-09-17T23:02:02.114 INFO:teuthology.orchestra.run.smithi006.stdout:(8/32): librgw2-19.1.1-330.gc2ddf972.el9.x86_64 7.7 MB/s | 5.3 MB 00:00 2024-09-17T23:02:02.139 INFO:teuthology.orchestra.run.smithi006.stdout:(9/32): python3-cephfs-19.1.1-330.gc2ddf972.el9 1.3 MB/s | 162 kB 00:00 2024-09-17T23:02:02.198 INFO:teuthology.orchestra.run.smithi006.stdout:(10/32): python3-rados-19.1.1-330.gc2ddf972.el9 3.8 MB/s | 320 kB 00:00 2024-09-17T23:02:02.232 INFO:teuthology.orchestra.run.smithi006.stdout:(11/32): python3-rbd-19.1.1-330.gc2ddf972.el9.x 3.2 MB/s | 300 kB 00:00 2024-09-17T23:02:02.282 INFO:teuthology.orchestra.run.smithi006.stdout:(12/32): python3-rgw-19.1.1-330.gc2ddf972.el9.x 1.2 MB/s | 99 kB 00:00 2024-09-17T23:02:02.424 INFO:teuthology.orchestra.run.smithi006.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 557 kB/s | 107 kB 00:00 2024-09-17T23:02:02.457 INFO:teuthology.orchestra.run.smithi006.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 914 kB/s | 160 kB 00:00 2024-09-17T23:02:02.478 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-09-17T23:02:02.491 INFO:teuthology.orchestra.run.smithi006.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 678 kB/s | 45 kB 00:00 2024-09-17T23:02:02.498 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-17T23:02:02.498 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repository Size 2024-09-17T23:02:02.498 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-17T23:02:02.498 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-09-17T23:02:02.498 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-radosgw x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 11 M 2024-09-17T23:02:02.498 INFO:teuthology.orchestra.run.smithi038.stdout:Upgrading: 2024-09-17T23:02:02.498 INFO:teuthology.orchestra.run.smithi038.stdout: librados2 x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 3.4 M 2024-09-17T23:02:02.498 INFO:teuthology.orchestra.run.smithi038.stdout: librbd1 x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 3.2 M 2024-09-17T23:02:02.498 INFO:teuthology.orchestra.run.smithi038.stdout:Installing dependencies: 2024-09-17T23:02:02.498 INFO:teuthology.orchestra.run.smithi038.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-17T23:02:02.499 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-base x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 5.4 M 2024-09-17T23:02:02.499 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-common x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 21 M 2024-09-17T23:02:02.499 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-selinux x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 25 k 2024-09-17T23:02:02.499 INFO:teuthology.orchestra.run.smithi038.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-17T23:02:02.499 INFO:teuthology.orchestra.run.smithi038.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-17T23:02:02.499 INFO:teuthology.orchestra.run.smithi038.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-17T23:02:02.499 INFO:teuthology.orchestra.run.smithi038.stdout: libcephfs2 x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 751 k 2024-09-17T23:02:02.499 INFO:teuthology.orchestra.run.smithi038.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-17T23:02:02.499 INFO:teuthology.orchestra.run.smithi038.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-17T23:02:02.499 INFO:teuthology.orchestra.run.smithi038.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-17T23:02:02.499 INFO:teuthology.orchestra.run.smithi038.stdout: libradosstriper1 x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 506 k 2024-09-17T23:02:02.499 INFO:teuthology.orchestra.run.smithi038.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-17T23:02:02.499 INFO:teuthology.orchestra.run.smithi038.stdout: librgw2 x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 5.3 M 2024-09-17T23:02:02.499 INFO:teuthology.orchestra.run.smithi038.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-17T23:02:02.500 INFO:teuthology.orchestra.run.smithi038.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-17T23:02:02.500 INFO:teuthology.orchestra.run.smithi038.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-17T23:02:02.500 INFO:teuthology.orchestra.run.smithi038.stdout: python3-ceph-argparse x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 45 k 2024-09-17T23:02:02.500 INFO:teuthology.orchestra.run.smithi038.stdout: python3-ceph-common x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 136 k 2024-09-17T23:02:02.500 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cephfs x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 162 k 2024-09-17T23:02:02.500 INFO:teuthology.orchestra.run.smithi038.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-17T23:02:02.500 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rados x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 320 k 2024-09-17T23:02:02.500 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rbd x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 300 k 2024-09-17T23:02:02.500 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rgw x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 99 k 2024-09-17T23:02:02.500 INFO:teuthology.orchestra.run.smithi038.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-17T23:02:02.500 INFO:teuthology.orchestra.run.smithi038.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-17T23:02:02.500 INFO:teuthology.orchestra.run.smithi038.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-17T23:02:02.500 INFO:teuthology.orchestra.run.smithi038.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-17T23:02:02.501 INFO:teuthology.orchestra.run.smithi038.stdout:Installing weak dependencies: 2024-09-17T23:02:02.501 INFO:teuthology.orchestra.run.smithi038.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-17T23:02:02.501 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:02:02.501 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-09-17T23:02:02.501 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-17T23:02:02.501 INFO:teuthology.orchestra.run.smithi038.stdout:Install 30 Packages 2024-09-17T23:02:02.501 INFO:teuthology.orchestra.run.smithi038.stdout:Upgrade 2 Packages 2024-09-17T23:02:02.501 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:02:02.502 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 60 M 2024-09-17T23:02:02.502 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-09-17T23:02:02.708 INFO:teuthology.orchestra.run.smithi006.stdout:(16/32): ceph-common-19.1.1-330.gc2ddf972.el9.x 7.9 MB/s | 21 MB 00:02 2024-09-17T23:02:02.741 INFO:teuthology.orchestra.run.smithi006.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.1 MB/s | 292 kB 00:00 2024-09-17T23:02:02.767 INFO:teuthology.orchestra.run.smithi006.stdout:(18/32): librdkafka-1.6.1-102.el9.x86_64.rpm 2.1 MB/s | 662 kB 00:00 2024-09-17T23:02:02.800 INFO:teuthology.orchestra.run.smithi006.stdout:(19/32): qatlib-24.02.0-1.el9.x86_64.rpm 3.7 MB/s | 221 kB 00:00 2024-09-17T23:02:02.826 INFO:teuthology.orchestra.run.smithi006.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 613 kB/s | 36 kB 00:00 2024-09-17T23:02:02.851 INFO:teuthology.orchestra.run.smithi006.stdout:(21/32): python3-prettytable-0.7.2-27.el9.noarc 296 kB/s | 42 kB 00:00 2024-09-17T23:02:02.876 INFO:teuthology.orchestra.run.smithi006.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 627 kB/s | 47 kB 00:00 2024-09-17T23:02:02.943 INFO:teuthology.orchestra.run.smithi006.stdout:(23/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 506 kB/s | 26 kB 00:00 2024-09-17T23:02:02.977 INFO:teuthology.orchestra.run.smithi006.stdout:(24/32): gperftools-libs-2.9.1-3.el9.x86_64.rpm 2.0 MB/s | 308 kB 00:00 2024-09-17T23:02:03.002 INFO:teuthology.orchestra.run.smithi006.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 828 kB/s | 49 kB 00:00 2024-09-17T23:02:03.027 INFO:teuthology.orchestra.run.smithi006.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-09-17T23:02:03.094 INFO:teuthology.orchestra.run.smithi006.stdout:(27/32): re2-20211101-20.el9.x86_64.rpm 2.8 MB/s | 191 kB 00:00 2024-09-17T23:02:03.228 INFO:teuthology.orchestra.run.smithi006.stdout:(28/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 3.6 MB/s | 839 kB 00:00 2024-09-17T23:02:03.387 INFO:teuthology.orchestra.run.smithi006.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 5.4 MB/s | 1.6 MB 00:00 2024-09-17T23:02:03.487 INFO:teuthology.orchestra.run.smithi006.stdout:(30/32): librados2-19.1.1-330.gc2ddf972.el9.x86 13 MB/s | 3.4 MB 00:00 2024-09-17T23:02:03.571 INFO:teuthology.orchestra.run.smithi006.stdout:(31/32): libarrow-9.0.0-11.el9.x86_64.rpm 6.1 MB/s | 4.4 MB 00:00 2024-09-17T23:02:03.591 INFO:teuthology.orchestra.run.smithi038.stdout:(1/32): ceph-base-19.1.1-330.gc2ddf972.el9.x86_ 8.1 MB/s | 5.4 MB 00:00 2024-09-17T23:02:03.641 INFO:teuthology.orchestra.run.smithi038.stdout:(2/32): ceph-selinux-19.1.1-330.gc2ddf972.el9.x 505 kB/s | 25 kB 00:00 2024-09-17T23:02:03.671 INFO:teuthology.orchestra.run.smithi006.stdout:(32/32): librbd1-19.1.1-330.gc2ddf972.el9.x86_6 11 MB/s | 3.2 MB 00:00 2024-09-17T23:02:03.676 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-09-17T23:02:03.676 INFO:teuthology.orchestra.run.smithi006.stdout:Total 16 MB/s | 60 MB 00:03 2024-09-17T23:02:03.725 INFO:teuthology.orchestra.run.smithi038.stdout:(3/32): libcephfs2-19.1.1-330.gc2ddf972.el9.x86 8.8 MB/s | 751 kB 00:00 2024-09-17T23:02:03.785 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-09-17T23:02:03.809 INFO:teuthology.orchestra.run.smithi038.stdout:(4/32): libradosstriper1-19.1.1-330.gc2ddf972.e 5.9 MB/s | 506 kB 00:00 2024-09-17T23:02:03.825 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-09-17T23:02:03.825 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-09-17T23:02:03.993 INFO:teuthology.orchestra.run.smithi038.stdout:(5/32): ceph-radosgw-19.1.1-330.gc2ddf972.el9.x 9.9 MB/s | 11 MB 00:01 2024-09-17T23:02:04.034 INFO:teuthology.orchestra.run.smithi038.stdout:(6/32): python3-ceph-argparse-19.1.1-330.gc2ddf 1.1 MB/s | 45 kB 00:00 2024-09-17T23:02:04.101 INFO:teuthology.orchestra.run.smithi038.stdout:(7/32): python3-ceph-common-19.1.1-330.gc2ddf97 2.0 MB/s | 136 kB 00:00 2024-09-17T23:02:04.168 INFO:teuthology.orchestra.run.smithi038.stdout:(8/32): python3-cephfs-19.1.1-330.gc2ddf972.el9 2.4 MB/s | 162 kB 00:00 2024-09-17T23:02:04.360 INFO:teuthology.orchestra.run.smithi038.stdout:(9/32): librgw2-19.1.1-330.gc2ddf972.el9.x86_64 9.7 MB/s | 5.3 MB 00:00 2024-09-17T23:02:04.385 INFO:teuthology.orchestra.run.smithi038.stdout:(10/32): python3-rados-19.1.1-330.gc2ddf972.el9 1.4 MB/s | 320 kB 00:00 2024-09-17T23:02:04.410 INFO:teuthology.orchestra.run.smithi038.stdout:(11/32): python3-rbd-19.1.1-330.gc2ddf972.el9.x 5.9 MB/s | 300 kB 00:00 2024-09-17T23:02:04.436 INFO:teuthology.orchestra.run.smithi038.stdout:(12/32): python3-rgw-19.1.1-330.gc2ddf972.el9.x 1.9 MB/s | 99 kB 00:00 2024-09-17T23:02:04.524 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-09-17T23:02:04.525 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-09-17T23:02:04.645 INFO:teuthology.orchestra.run.smithi038.stdout:(13/32): ceph-common-19.1.1-330.gc2ddf972.el9.x 12 MB/s | 21 MB 00:01 2024-09-17T23:02:04.753 INFO:teuthology.orchestra.run.smithi038.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 418 kB/s | 45 kB 00:00 2024-09-17T23:02:04.778 INFO:teuthology.orchestra.run.smithi038.stdout:(15/32): boost-program-options-1.75.0-8.el9.x86 291 kB/s | 107 kB 00:00 2024-09-17T23:02:04.804 INFO:teuthology.orchestra.run.smithi038.stdout:(16/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 436 kB/s | 160 kB 00:00 2024-09-17T23:02:04.846 INFO:teuthology.orchestra.run.smithi038.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 1.0 MB/s | 42 kB 00:00 2024-09-17T23:02:04.871 INFO:teuthology.orchestra.run.smithi038.stdout:(18/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.1 MB/s | 292 kB 00:00 2024-09-17T23:02:04.896 INFO:teuthology.orchestra.run.smithi038.stdout:(19/32): librdkafka-1.6.1-102.el9.x86_64.rpm 4.5 MB/s | 662 kB 00:00 2024-09-17T23:02:04.921 INFO:teuthology.orchestra.run.smithi038.stdout:(20/32): qatlib-24.02.0-1.el9.x86_64.rpm 2.9 MB/s | 221 kB 00:00 2024-09-17T23:02:04.947 INFO:teuthology.orchestra.run.smithi038.stdout:(21/32): qatlib-service-24.02.0-1.el9.x86_64.rp 476 kB/s | 36 kB 00:00 2024-09-17T23:02:05.027 INFO:teuthology.orchestra.run.smithi038.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 364 kB/s | 47 kB 00:00 2024-09-17T23:02:05.097 INFO:teuthology.orchestra.run.smithi038.stdout:(23/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 372 kB/s | 26 kB 00:00 2024-09-17T23:02:05.123 INFO:teuthology.orchestra.run.smithi038.stdout:(24/32): gperftools-libs-2.9.1-3.el9.x86_64.rpm 1.7 MB/s | 308 kB 00:00 2024-09-17T23:02:05.148 INFO:teuthology.orchestra.run.smithi038.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 965 kB/s | 49 kB 00:00 2024-09-17T23:02:05.173 INFO:teuthology.orchestra.run.smithi038.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-09-17T23:02:05.257 INFO:teuthology.orchestra.run.smithi038.stdout:(27/32): re2-20211101-20.el9.x86_64.rpm 2.2 MB/s | 191 kB 00:00 2024-09-17T23:02:05.307 INFO:teuthology.orchestra.run.smithi038.stdout:(28/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 5.2 MB/s | 839 kB 00:00 2024-09-17T23:02:05.532 INFO:teuthology.orchestra.run.smithi038.stdout:(29/32): librados2-19.1.1-330.gc2ddf972.el9.x86 15 MB/s | 3.4 MB 00:00 2024-09-17T23:02:05.574 INFO:teuthology.orchestra.run.smithi038.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 5.0 MB/s | 1.6 MB 00:00 2024-09-17T23:02:05.708 INFO:teuthology.orchestra.run.smithi038.stdout:(31/32): librbd1-19.1.1-330.gc2ddf972.el9.x86_6 18 MB/s | 3.2 MB 00:00 2024-09-17T23:02:05.783 INFO:teuthology.orchestra.run.smithi038.stdout:(32/32): libarrow-9.0.0-11.el9.x86_64.rpm 5.3 MB/s | 4.4 MB 00:00 2024-09-17T23:02:05.788 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-09-17T23:02:05.788 INFO:teuthology.orchestra.run.smithi038.stdout:Total 18 MB/s | 60 MB 00:03 2024-09-17T23:02:05.901 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-09-17T23:02:05.943 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-09-17T23:02:05.943 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-09-17T23:02:06.043 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-09-17T23:02:06.141 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-09-17T23:02:06.176 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-09-17T23:02:06.209 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-09-17T23:02:06.232 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-17T23:02:06.262 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-17T23:02:06.460 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-17T23:02:06.521 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-17T23:02:06.642 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-09-17T23:02:06.642 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-09-17T23:02:06.888 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-09-17T23:02:07.002 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : librados2-2:19.1.1-330.gc2ddf972.el9.x86_64 7/34 2024-09-17T23:02:07.096 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: librados2-2:19.1.1-330.gc2ddf972.el9.x86_64 7/34 2024-09-17T23:02:07.202 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-rados-2:19.1.1-330.gc2ddf972.el9.x86_64 8/34 2024-09-17T23:02:07.237 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libcephfs2-2:19.1.1-330.gc2ddf972.el9.x86_64 9/34 2024-09-17T23:02:07.300 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: libcephfs2-2:19.1.1-330.gc2ddf972.el9.x86_64 9/34 2024-09-17T23:02:07.329 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-09-17T23:02:07.373 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-ceph-argparse-2:19.1.1-330.gc2ddf972.el9.x 11/34 2024-09-17T23:02:07.451 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-cephfs-2:19.1.1-330.gc2ddf972.el9.x86_64 12/34 2024-09-17T23:02:07.481 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libradosstriper1-2:19.1.1-330.gc2ddf972.el9.x86_64 13/34 2024-09-17T23:02:07.546 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: libradosstriper1-2:19.1.1-330.gc2ddf972.el9.x86_64 13/34 2024-09-17T23:02:07.580 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-09-17T23:02:07.699 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/34 2024-09-17T23:02:07.731 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-09-17T23:02:07.778 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-09-17T23:02:07.813 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/34 2024-09-17T23:02:07.842 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-09-17T23:02:07.875 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-09-17T23:02:08.153 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-09-17T23:02:08.202 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-09-17T23:02:08.253 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-09-17T23:02:08.271 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : librbd1-2:19.1.1-330.gc2ddf972.el9.x86_64 22/34 2024-09-17T23:02:08.288 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-09-17T23:02:08.313 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-09-17T23:02:08.344 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-17T23:02:08.362 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: librbd1-2:19.1.1-330.gc2ddf972.el9.x86_64 22/34 2024-09-17T23:02:08.373 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-17T23:02:08.392 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-rbd-2:19.1.1-330.gc2ddf972.el9.x86_64 23/34 2024-09-17T23:02:08.428 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-09-17T23:02:08.568 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-17T23:02:08.627 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-17T23:02:08.936 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-09-17T23:02:08.965 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : librgw2-2:19.1.1-330.gc2ddf972.el9.x86_64 26/34 2024-09-17T23:02:09.030 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-09-17T23:02:09.037 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: librgw2-2:19.1.1-330.gc2ddf972.el9.x86_64 26/34 2024-09-17T23:02:09.106 INFO:teuthology.orchestra.run.smithi038.stdout: Upgrading : librados2-2:19.1.1-330.gc2ddf972.el9.x86_64 7/34 2024-09-17T23:02:09.117 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-rgw-2:19.1.1-330.gc2ddf972.el9.x86_64 27/34 2024-09-17T23:02:09.159 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-ceph-common-2:19.1.1-330.gc2ddf972.el9.x86 28/34 2024-09-17T23:02:09.212 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: librados2-2:19.1.1-330.gc2ddf972.el9.x86_64 7/34 2024-09-17T23:02:09.326 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-rados-2:19.1.1-330.gc2ddf972.el9.x86_64 8/34 2024-09-17T23:02:09.358 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libcephfs2-2:19.1.1-330.gc2ddf972.el9.x86_64 9/34 2024-09-17T23:02:09.421 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: libcephfs2-2:19.1.1-330.gc2ddf972.el9.x86_64 9/34 2024-09-17T23:02:09.458 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-09-17T23:02:09.502 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-ceph-argparse-2:19.1.1-330.gc2ddf972.el9.x 11/34 2024-09-17T23:02:09.573 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cephfs-2:19.1.1-330.gc2ddf972.el9.x86_64 12/34 2024-09-17T23:02:09.602 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libradosstriper1-2:19.1.1-330.gc2ddf972.el9.x86_64 13/34 2024-09-17T23:02:09.658 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: libradosstriper1-2:19.1.1-330.gc2ddf972.el9.x86_64 13/34 2024-09-17T23:02:09.692 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-09-17T23:02:09.812 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/34 2024-09-17T23:02:09.852 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-09-17T23:02:09.904 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-09-17T23:02:09.933 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/34 2024-09-17T23:02:09.963 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-09-17T23:02:09.996 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-09-17T23:02:10.323 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-09-17T23:02:10.366 INFO:teuthology.orchestra.run.smithi038.stdout: Upgrading : librbd1-2:19.1.1-330.gc2ddf972.el9.x86_64 22/34 2024-09-17T23:02:10.458 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: librbd1-2:19.1.1-330.gc2ddf972.el9.x86_64 22/34 2024-09-17T23:02:10.488 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-rbd-2:19.1.1-330.gc2ddf972.el9.x86_64 23/34 2024-09-17T23:02:10.532 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-09-17T23:02:11.048 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-09-17T23:02:11.077 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : librgw2-2:19.1.1-330.gc2ddf972.el9.x86_64 26/34 2024-09-17T23:02:11.132 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: librgw2-2:19.1.1-330.gc2ddf972.el9.x86_64 26/34 2024-09-17T23:02:11.214 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-rgw-2:19.1.1-330.gc2ddf972.el9.x86_64 27/34 2024-09-17T23:02:11.246 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-ceph-common-2:19.1.1-330.gc2ddf972.el9.x86 28/34 2024-09-17T23:02:11.525 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-common-2:19.1.1-330.gc2ddf972.el9.x86_64 29/34 2024-09-17T23:02:11.564 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-common-2:19.1.1-330.gc2ddf972.el9.x86_64 29/34 2024-09-17T23:02:12.173 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-common-2:19.1.1-330.gc2ddf972.el9.x86_64 29/34 2024-09-17T23:02:12.217 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-base-2:19.1.1-330.gc2ddf972.el9.x86_64 30/34 2024-09-17T23:02:12.339 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-base-2:19.1.1-330.gc2ddf972.el9.x86_64 30/34 2024-09-17T23:02:12.339 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-17T23:02:12.339 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-17T23:02:12.339 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:02:12.377 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-selinux-2:19.1.1-330.gc2ddf972.el9.x86_64 31/34 2024-09-17T23:02:13.617 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-common-2:19.1.1-330.gc2ddf972.el9.x86_64 29/34 2024-09-17T23:02:13.650 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-common-2:19.1.1-330.gc2ddf972.el9.x86_64 29/34 2024-09-17T23:02:14.239 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-common-2:19.1.1-330.gc2ddf972.el9.x86_64 29/34 2024-09-17T23:02:14.279 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-base-2:19.1.1-330.gc2ddf972.el9.x86_64 30/34 2024-09-17T23:02:14.371 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-base-2:19.1.1-330.gc2ddf972.el9.x86_64 30/34 2024-09-17T23:02:14.371 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-17T23:02:14.372 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-17T23:02:14.372 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:02:14.405 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-selinux-2:19.1.1-330.gc2ddf972.el9.x86_64 31/34 2024-09-17T23:02:22.883 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-selinux-2:19.1.1-330.gc2ddf972.el9.x86_64 31/34 2024-09-17T23:02:22.884 INFO:teuthology.orchestra.run.smithi006.stdout:skipping the directory /sys 2024-09-17T23:02:22.884 INFO:teuthology.orchestra.run.smithi006.stdout:skipping the directory /proc 2024-09-17T23:02:22.884 INFO:teuthology.orchestra.run.smithi006.stdout:skipping the directory /mnt 2024-09-17T23:02:22.884 INFO:teuthology.orchestra.run.smithi006.stdout:skipping the directory /var/tmp 2024-09-17T23:02:22.884 INFO:teuthology.orchestra.run.smithi006.stdout:skipping the directory /home 2024-09-17T23:02:22.884 INFO:teuthology.orchestra.run.smithi006.stdout:skipping the directory /root 2024-09-17T23:02:22.884 INFO:teuthology.orchestra.run.smithi006.stdout:skipping the directory /tmp 2024-09-17T23:02:22.884 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:02:23.885 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-radosgw-2:19.1.1-330.gc2ddf972.el9.x86_64 32/34 2024-09-17T23:02:23.912 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-radosgw-2:19.1.1-330.gc2ddf972.el9.x86_64 32/34 2024-09-17T23:02:23.912 INFO:teuthology.orchestra.run.smithi006.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T23:02:23.912 INFO:teuthology.orchestra.run.smithi006.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-17T23:02:23.912 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-17T23:02:23.912 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-17T23:02:23.912 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:02:23.914 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-17T23:02:23.996 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-17T23:02:23.996 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-17T23:02:24.723 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-selinux-2:19.1.1-330.gc2ddf972.el9.x86_64 31/34 2024-09-17T23:02:24.723 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /sys 2024-09-17T23:02:24.723 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /proc 2024-09-17T23:02:24.723 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /mnt 2024-09-17T23:02:24.723 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /var/tmp 2024-09-17T23:02:24.723 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /home 2024-09-17T23:02:24.723 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /root 2024-09-17T23:02:24.723 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /tmp 2024-09-17T23:02:24.723 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:02:25.246 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-17T23:02:25.246 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-base-2:19.1.1-330.gc2ddf972.el9.x86_64 1/34 2024-09-17T23:02:25.246 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-common-2:19.1.1-330.gc2ddf972.el9.x86_64 2/34 2024-09-17T23:02:25.246 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-radosgw-2:19.1.1-330.gc2ddf972.el9.x86_64 3/34 2024-09-17T23:02:25.246 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-selinux-2:19.1.1-330.gc2ddf972.el9.x86_64 4/34 2024-09-17T23:02:25.246 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libcephfs2-2:19.1.1-330.gc2ddf972.el9.x86_64 5/34 2024-09-17T23:02:25.246 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libradosstriper1-2:19.1.1-330.gc2ddf972.el9.x86_64 6/34 2024-09-17T23:02:25.246 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librgw2-2:19.1.1-330.gc2ddf972.el9.x86_64 7/34 2024-09-17T23:02:25.247 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-ceph-argparse-2:19.1.1-330.gc2ddf972.el9.x 8/34 2024-09-17T23:02:25.247 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-ceph-common-2:19.1.1-330.gc2ddf972.el9.x86 9/34 2024-09-17T23:02:25.247 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-cephfs-2:19.1.1-330.gc2ddf972.el9.x86_64 10/34 2024-09-17T23:02:25.247 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-rados-2:19.1.1-330.gc2ddf972.el9.x86_64 11/34 2024-09-17T23:02:25.247 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-rbd-2:19.1.1-330.gc2ddf972.el9.x86_64 12/34 2024-09-17T23:02:25.247 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-rgw-2:19.1.1-330.gc2ddf972.el9.x86_64 13/34 2024-09-17T23:02:25.247 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-09-17T23:02:25.247 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-09-17T23:02:25.247 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-09-17T23:02:25.247 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-09-17T23:02:25.247 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-09-17T23:02:25.247 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-09-17T23:02:25.247 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-09-17T23:02:25.248 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-09-17T23:02:25.248 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-09-17T23:02:25.248 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 23/34 2024-09-17T23:02:25.248 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-09-17T23:02:25.248 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-09-17T23:02:25.248 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-09-17T23:02:25.248 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-09-17T23:02:25.248 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-09-17T23:02:25.248 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : re2-1:20211101-20.el9.x86_64 29/34 2024-09-17T23:02:25.248 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-09-17T23:02:25.248 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librados2-2:19.1.1-330.gc2ddf972.el9.x86_64 31/34 2024-09-17T23:02:25.248 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-09-17T23:02:25.248 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librbd1-2:19.1.1-330.gc2ddf972.el9.x86_64 33/34 2024-09-17T23:02:25.721 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-radosgw-2:19.1.1-330.gc2ddf972.el9.x86_64 32/34 2024-09-17T23:02:25.748 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-radosgw-2:19.1.1-330.gc2ddf972.el9.x86_64 32/34 2024-09-17T23:02:25.749 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T23:02:25.749 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-17T23:02:25.749 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-17T23:02:25.749 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-17T23:02:25.749 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:02:25.750 INFO:teuthology.orchestra.run.smithi038.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-17T23:02:25.828 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-17T23:02:25.829 INFO:teuthology.orchestra.run.smithi038.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-17T23:02:26.235 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-09-17T23:02:26.235 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:02:26.235 INFO:teuthology.orchestra.run.smithi006.stdout:Upgraded: 2024-09-17T23:02:26.235 INFO:teuthology.orchestra.run.smithi006.stdout: librados2-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:02:26.235 INFO:teuthology.orchestra.run.smithi006.stdout: librbd1-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:02:26.236 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-09-17T23:02:26.236 INFO:teuthology.orchestra.run.smithi006.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-17T23:02:26.236 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-base-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:02:26.236 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-common-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:02:26.236 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-radosgw-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:02:26.236 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-selinux-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:02:26.236 INFO:teuthology.orchestra.run.smithi006.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-17T23:02:26.236 INFO:teuthology.orchestra.run.smithi006.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-17T23:02:26.236 INFO:teuthology.orchestra.run.smithi006.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-17T23:02:26.236 INFO:teuthology.orchestra.run.smithi006.stdout: libcephfs2-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:02:26.237 INFO:teuthology.orchestra.run.smithi006.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-17T23:02:26.237 INFO:teuthology.orchestra.run.smithi006.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-17T23:02:26.237 INFO:teuthology.orchestra.run.smithi006.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-17T23:02:26.237 INFO:teuthology.orchestra.run.smithi006.stdout: libradosstriper1-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:02:26.237 INFO:teuthology.orchestra.run.smithi006.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-17T23:02:26.237 INFO:teuthology.orchestra.run.smithi006.stdout: librgw2-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:02:26.237 INFO:teuthology.orchestra.run.smithi006.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-17T23:02:26.237 INFO:teuthology.orchestra.run.smithi006.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-17T23:02:26.237 INFO:teuthology.orchestra.run.smithi006.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-17T23:02:26.237 INFO:teuthology.orchestra.run.smithi006.stdout: python3-ceph-argparse-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:02:26.237 INFO:teuthology.orchestra.run.smithi006.stdout: python3-ceph-common-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:02:26.237 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cephfs-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:02:26.237 INFO:teuthology.orchestra.run.smithi006.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-17T23:02:26.238 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rados-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:02:26.238 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rbd-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:02:26.238 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rgw-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:02:26.238 INFO:teuthology.orchestra.run.smithi006.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-17T23:02:26.238 INFO:teuthology.orchestra.run.smithi006.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-17T23:02:26.238 INFO:teuthology.orchestra.run.smithi006.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-17T23:02:26.238 INFO:teuthology.orchestra.run.smithi006.stdout: re2-1:20211101-20.el9.x86_64 2024-09-17T23:02:26.238 INFO:teuthology.orchestra.run.smithi006.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-17T23:02:26.238 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:02:26.238 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-09-17T23:02:26.650 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install ceph-test 2024-09-17T23:02:27.059 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-17T23:02:27.059 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-base-2:19.1.1-330.gc2ddf972.el9.x86_64 1/34 2024-09-17T23:02:27.059 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-common-2:19.1.1-330.gc2ddf972.el9.x86_64 2/34 2024-09-17T23:02:27.059 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-radosgw-2:19.1.1-330.gc2ddf972.el9.x86_64 3/34 2024-09-17T23:02:27.059 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-selinux-2:19.1.1-330.gc2ddf972.el9.x86_64 4/34 2024-09-17T23:02:27.059 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libcephfs2-2:19.1.1-330.gc2ddf972.el9.x86_64 5/34 2024-09-17T23:02:27.059 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libradosstriper1-2:19.1.1-330.gc2ddf972.el9.x86_64 6/34 2024-09-17T23:02:27.059 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librgw2-2:19.1.1-330.gc2ddf972.el9.x86_64 7/34 2024-09-17T23:02:27.060 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-ceph-argparse-2:19.1.1-330.gc2ddf972.el9.x 8/34 2024-09-17T23:02:27.060 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-ceph-common-2:19.1.1-330.gc2ddf972.el9.x86 9/34 2024-09-17T23:02:27.060 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cephfs-2:19.1.1-330.gc2ddf972.el9.x86_64 10/34 2024-09-17T23:02:27.060 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-rados-2:19.1.1-330.gc2ddf972.el9.x86_64 11/34 2024-09-17T23:02:27.060 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-rbd-2:19.1.1-330.gc2ddf972.el9.x86_64 12/34 2024-09-17T23:02:27.060 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-rgw-2:19.1.1-330.gc2ddf972.el9.x86_64 13/34 2024-09-17T23:02:27.060 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-09-17T23:02:27.060 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-09-17T23:02:27.060 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-09-17T23:02:27.060 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-09-17T23:02:27.060 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-09-17T23:02:27.060 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-09-17T23:02:27.060 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-09-17T23:02:27.060 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-09-17T23:02:27.061 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-09-17T23:02:27.061 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 23/34 2024-09-17T23:02:27.061 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-09-17T23:02:27.061 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-09-17T23:02:27.061 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-09-17T23:02:27.061 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-09-17T23:02:27.061 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-09-17T23:02:27.061 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : re2-1:20211101-20.el9.x86_64 29/34 2024-09-17T23:02:27.061 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-09-17T23:02:27.061 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librados2-2:19.1.1-330.gc2ddf972.el9.x86_64 31/34 2024-09-17T23:02:27.061 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-09-17T23:02:27.061 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librbd1-2:19.1.1-330.gc2ddf972.el9.x86_64 33/34 2024-09-17T23:02:27.250 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:00:30 ago on Tue 17 Sep 2024 11:01:57 PM UTC. 2024-09-17T23:02:27.389 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-09-17T23:02:27.389 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-09-17T23:02:27.389 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repository Size 2024-09-17T23:02:27.389 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-09-17T23:02:27.389 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-09-17T23:02:27.389 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-test x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 49 M 2024-09-17T23:02:27.390 INFO:teuthology.orchestra.run.smithi006.stdout:Installing dependencies: 2024-09-17T23:02:27.390 INFO:teuthology.orchestra.run.smithi006.stdout: libcephsqlite x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 163 k 2024-09-17T23:02:27.390 INFO:teuthology.orchestra.run.smithi006.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-17T23:02:27.390 INFO:teuthology.orchestra.run.smithi006.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-17T23:02:27.390 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:02:27.390 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-09-17T23:02:27.390 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-09-17T23:02:27.390 INFO:teuthology.orchestra.run.smithi006.stdout:Install 4 Packages 2024-09-17T23:02:27.390 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:02:27.391 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 49 M 2024-09-17T23:02:27.391 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 209 M 2024-09-17T23:02:27.391 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-09-17T23:02:27.644 INFO:teuthology.orchestra.run.smithi006.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 2.3 MB/s | 304 kB 00:00 2024-09-17T23:02:27.686 INFO:teuthology.orchestra.run.smithi006.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 64 kB 00:00 2024-09-17T23:02:27.736 INFO:teuthology.orchestra.run.smithi006.stdout:(3/4): libcephsqlite-19.1.1-330.gc2ddf972.el9.x 744 kB/s | 163 kB 00:00 2024-09-17T23:02:27.949 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-09-17T23:02:27.949 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:02:27.949 INFO:teuthology.orchestra.run.smithi038.stdout:Upgraded: 2024-09-17T23:02:27.949 INFO:teuthology.orchestra.run.smithi038.stdout: librados2-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:02:27.949 INFO:teuthology.orchestra.run.smithi038.stdout: librbd1-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:02:27.949 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-09-17T23:02:27.949 INFO:teuthology.orchestra.run.smithi038.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-17T23:02:27.949 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-base-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:02:27.949 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-common-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:02:27.950 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-radosgw-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:02:27.950 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-selinux-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:02:27.950 INFO:teuthology.orchestra.run.smithi038.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-17T23:02:27.950 INFO:teuthology.orchestra.run.smithi038.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-17T23:02:27.950 INFO:teuthology.orchestra.run.smithi038.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-17T23:02:27.950 INFO:teuthology.orchestra.run.smithi038.stdout: libcephfs2-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:02:27.950 INFO:teuthology.orchestra.run.smithi038.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-17T23:02:27.950 INFO:teuthology.orchestra.run.smithi038.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-17T23:02:27.950 INFO:teuthology.orchestra.run.smithi038.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-17T23:02:27.950 INFO:teuthology.orchestra.run.smithi038.stdout: libradosstriper1-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:02:27.950 INFO:teuthology.orchestra.run.smithi038.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-17T23:02:27.950 INFO:teuthology.orchestra.run.smithi038.stdout: librgw2-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:02:27.950 INFO:teuthology.orchestra.run.smithi038.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-17T23:02:27.951 INFO:teuthology.orchestra.run.smithi038.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-17T23:02:27.951 INFO:teuthology.orchestra.run.smithi038.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-17T23:02:27.951 INFO:teuthology.orchestra.run.smithi038.stdout: python3-ceph-argparse-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:02:27.951 INFO:teuthology.orchestra.run.smithi038.stdout: python3-ceph-common-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:02:27.951 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cephfs-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:02:27.951 INFO:teuthology.orchestra.run.smithi038.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-17T23:02:27.951 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rados-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:02:27.951 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rbd-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:02:27.951 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rgw-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:02:27.951 INFO:teuthology.orchestra.run.smithi038.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-17T23:02:27.951 INFO:teuthology.orchestra.run.smithi038.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-17T23:02:27.951 INFO:teuthology.orchestra.run.smithi038.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-17T23:02:27.951 INFO:teuthology.orchestra.run.smithi038.stdout: re2-1:20211101-20.el9.x86_64 2024-09-17T23:02:27.951 INFO:teuthology.orchestra.run.smithi038.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-17T23:02:27.952 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:02:27.952 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-09-17T23:02:28.278 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-test 2024-09-17T23:02:28.880 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:00:28 ago on Tue 17 Sep 2024 11:02:00 PM UTC. 2024-09-17T23:02:29.029 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-09-17T23:02:29.029 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-17T23:02:29.029 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repository Size 2024-09-17T23:02:29.029 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-17T23:02:29.029 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-09-17T23:02:29.029 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-test x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 49 M 2024-09-17T23:02:29.029 INFO:teuthology.orchestra.run.smithi038.stdout:Installing dependencies: 2024-09-17T23:02:29.029 INFO:teuthology.orchestra.run.smithi038.stdout: libcephsqlite x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 163 k 2024-09-17T23:02:29.029 INFO:teuthology.orchestra.run.smithi038.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-17T23:02:29.029 INFO:teuthology.orchestra.run.smithi038.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-17T23:02:29.029 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:02:29.030 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-09-17T23:02:29.030 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-17T23:02:29.030 INFO:teuthology.orchestra.run.smithi038.stdout:Install 4 Packages 2024-09-17T23:02:29.030 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:02:29.030 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 49 M 2024-09-17T23:02:29.030 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 209 M 2024-09-17T23:02:29.030 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-09-17T23:02:29.061 INFO:teuthology.orchestra.run.smithi006.stdout:(4/4): ceph-test-19.1.1-330.gc2ddf972.el9.x86_6 32 MB/s | 49 MB 00:01 2024-09-17T23:02:29.063 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-09-17T23:02:29.063 INFO:teuthology.orchestra.run.smithi006.stdout:Total 29 MB/s | 49 MB 00:01 2024-09-17T23:02:29.091 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-09-17T23:02:29.114 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-09-17T23:02:29.114 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-09-17T23:02:29.255 INFO:teuthology.orchestra.run.smithi038.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 2.4 MB/s | 304 kB 00:00 2024-09-17T23:02:29.297 INFO:teuthology.orchestra.run.smithi038.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 64 kB 00:00 2024-09-17T23:02:29.364 INFO:teuthology.orchestra.run.smithi038.stdout:(3/4): libcephsqlite-19.1.1-330.gc2ddf972.el9.x 702 kB/s | 163 kB 00:00 2024-09-17T23:02:29.518 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-09-17T23:02:29.518 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-09-17T23:02:30.070 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-09-17T23:02:30.148 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-17T23:02:30.187 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-17T23:02:30.213 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libcephsqlite-2:19.1.1-330.gc2ddf972.el9.x86_64 3/4 2024-09-17T23:02:30.640 INFO:teuthology.orchestra.run.smithi038.stdout:(4/4): ceph-test-19.1.1-330.gc2ddf972.el9.x86_6 32 MB/s | 49 MB 00:01 2024-09-17T23:02:30.641 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-09-17T23:02:30.642 INFO:teuthology.orchestra.run.smithi038.stdout:Total 30 MB/s | 49 MB 00:01 2024-09-17T23:02:30.670 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-09-17T23:02:30.693 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-09-17T23:02:30.693 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-09-17T23:02:31.099 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-09-17T23:02:31.099 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-09-17T23:02:31.642 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-09-17T23:02:31.728 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-17T23:02:31.781 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-17T23:02:31.808 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libcephsqlite-2:19.1.1-330.gc2ddf972.el9.x86_64 3/4 2024-09-17T23:02:35.104 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: libcephsqlite-2:19.1.1-330.gc2ddf972.el9.x86_64 3/4 2024-09-17T23:02:35.182 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-test-2:19.1.1-330.gc2ddf972.el9.x86_64 4/4 2024-09-17T23:02:36.738 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: libcephsqlite-2:19.1.1-330.gc2ddf972.el9.x86_64 3/4 2024-09-17T23:02:36.825 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-test-2:19.1.1-330.gc2ddf972.el9.x86_64 4/4 2024-09-17T23:02:37.591 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-test-2:19.1.1-330.gc2ddf972.el9.x86_64 4/4 2024-09-17T23:02:37.591 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-test-2:19.1.1-330.gc2ddf972.el9.x86_64 1/4 2024-09-17T23:02:37.591 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libcephsqlite-2:19.1.1-330.gc2ddf972.el9.x86_64 2/4 2024-09-17T23:02:37.591 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-17T23:02:37.951 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-17T23:02:37.952 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:02:37.952 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-09-17T23:02:37.952 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-test-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:02:37.952 INFO:teuthology.orchestra.run.smithi006.stdout: libcephsqlite-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:02:37.952 INFO:teuthology.orchestra.run.smithi006.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-17T23:02:37.952 INFO:teuthology.orchestra.run.smithi006.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-17T23:02:37.952 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:02:37.952 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-09-17T23:02:38.202 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install ceph 2024-09-17T23:02:38.808 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:00:41 ago on Tue 17 Sep 2024 11:01:57 PM UTC. 2024-09-17T23:02:38.952 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-09-17T23:02:38.955 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-09-17T23:02:38.955 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repository Size 2024-09-17T23:02:38.955 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-09-17T23:02:38.955 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-09-17T23:02:38.955 INFO:teuthology.orchestra.run.smithi006.stdout: ceph x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 6.5 k 2024-09-17T23:02:38.955 INFO:teuthology.orchestra.run.smithi006.stdout:Installing dependencies: 2024-09-17T23:02:38.955 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mds x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 2.4 M 2024-09-17T23:02:38.955 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 1.5 M 2024-09-17T23:02:38.955 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-modules-core noarch 2:19.1.1-330.gc2ddf972.el9 ceph-noarch 249 k 2024-09-17T23:02:38.955 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mon x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 4.7 M 2024-09-17T23:02:38.955 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-osd x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 17 M 2024-09-17T23:02:38.956 INFO:teuthology.orchestra.run.smithi006.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-17T23:02:38.956 INFO:teuthology.orchestra.run.smithi006.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-17T23:02:38.956 INFO:teuthology.orchestra.run.smithi006.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-17T23:02:38.956 INFO:teuthology.orchestra.run.smithi006.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-17T23:02:38.956 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-17T23:02:38.956 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-17T23:02:38.956 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-17T23:02:38.956 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-17T23:02:38.956 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-17T23:02:38.956 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-17T23:02:38.956 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-17T23:02:38.957 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-17T23:02:38.957 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-17T23:02:38.957 INFO:teuthology.orchestra.run.smithi006.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-17T23:02:38.957 INFO:teuthology.orchestra.run.smithi006.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-17T23:02:38.957 INFO:teuthology.orchestra.run.smithi006.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-17T23:02:38.957 INFO:teuthology.orchestra.run.smithi006.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-17T23:02:38.957 INFO:teuthology.orchestra.run.smithi006.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-17T23:02:38.957 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-17T23:02:38.957 INFO:teuthology.orchestra.run.smithi006.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-17T23:02:38.957 INFO:teuthology.orchestra.run.smithi006.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-17T23:02:38.957 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-17T23:02:38.957 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-17T23:02:38.957 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-17T23:02:38.957 INFO:teuthology.orchestra.run.smithi006.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-17T23:02:38.958 INFO:teuthology.orchestra.run.smithi006.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-17T23:02:38.958 INFO:teuthology.orchestra.run.smithi006.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-17T23:02:38.958 INFO:teuthology.orchestra.run.smithi006.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-17T23:02:38.958 INFO:teuthology.orchestra.run.smithi006.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-17T23:02:38.958 INFO:teuthology.orchestra.run.smithi006.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-17T23:02:38.958 INFO:teuthology.orchestra.run.smithi006.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-17T23:02:38.958 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:02:38.958 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-09-17T23:02:38.958 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-09-17T23:02:38.958 INFO:teuthology.orchestra.run.smithi006.stdout:Install 37 Packages 2024-09-17T23:02:38.958 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:02:38.959 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 30 M 2024-09-17T23:02:38.959 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 108 M 2024-09-17T23:02:38.959 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-09-17T23:02:39.342 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-test-2:19.1.1-330.gc2ddf972.el9.x86_64 4/4 2024-09-17T23:02:39.342 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-test-2:19.1.1-330.gc2ddf972.el9.x86_64 1/4 2024-09-17T23:02:39.342 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libcephsqlite-2:19.1.1-330.gc2ddf972.el9.x86_64 2/4 2024-09-17T23:02:39.342 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-17T23:02:39.508 INFO:teuthology.orchestra.run.smithi006.stdout:(1/37): ceph-19.1.1-330.gc2ddf972.el9.x86_64.rp 50 kB/s | 6.5 kB 00:00 2024-09-17T23:02:39.662 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-17T23:02:39.662 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:02:39.662 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-09-17T23:02:39.662 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-test-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:02:39.662 INFO:teuthology.orchestra.run.smithi038.stdout: libcephsqlite-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:02:39.662 INFO:teuthology.orchestra.run.smithi038.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-17T23:02:39.662 INFO:teuthology.orchestra.run.smithi038.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-17T23:02:39.662 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:02:39.663 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-09-17T23:02:39.666 INFO:teuthology.orchestra.run.smithi006.stdout:(2/37): ceph-mgr-19.1.1-330.gc2ddf972.el9.x86_6 5.3 MB/s | 1.5 MB 00:00 2024-09-17T23:02:39.725 INFO:teuthology.orchestra.run.smithi006.stdout:(3/37): ceph-mds-19.1.1-330.gc2ddf972.el9.x86_6 6.9 MB/s | 2.4 MB 00:00 2024-09-17T23:02:39.834 INFO:teuthology.orchestra.run.smithi006.stdout:(4/37): ceph-mgr-modules-core-19.1.1-330.gc2ddf 2.3 MB/s | 249 kB 00:00 2024-09-17T23:02:39.909 INFO:teuthology.orchestra.run.smithi006.stdout:(5/37): ceph-mon-19.1.1-330.gc2ddf972.el9.x86_6 12 MB/s | 4.7 MB 00:00 2024-09-17T23:02:39.959 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph 2024-09-17T23:02:40.043 INFO:teuthology.orchestra.run.smithi006.stdout:(6/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.2 MB/s | 253 kB 00:00 2024-09-17T23:02:40.093 INFO:teuthology.orchestra.run.smithi006.stdout:(7/37): python3-ply-3.11-14.el9.noarch.rpm 2.1 MB/s | 106 kB 00:00 2024-09-17T23:02:40.276 INFO:teuthology.orchestra.run.smithi006.stdout:(8/37): ceph-osd-19.1.1-330.gc2ddf972.el9.x86_6 28 MB/s | 17 MB 00:00 2024-09-17T23:02:40.310 INFO:teuthology.orchestra.run.smithi006.stdout:(9/37): python3-pycparser-2.20-6.el9.noarch.rpm 623 kB/s | 135 kB 00:00 2024-09-17T23:02:40.402 INFO:teuthology.orchestra.run.smithi006.stdout:(10/37): python3-cryptography-36.0.1-4.el9.x86_ 2.5 MB/s | 1.2 MB 00:00 2024-09-17T23:02:40.427 INFO:teuthology.orchestra.run.smithi006.stdout:(11/37): python3-urllib3-1.26.5-6.el9.noarch.rp 1.8 MB/s | 215 kB 00:00 2024-09-17T23:02:40.469 INFO:teuthology.orchestra.run.smithi006.stdout:(12/37): python3-requests-2.25.1-8.el9.noarch.r 648 kB/s | 125 kB 00:00 2024-09-17T23:02:40.574 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:00:40 ago on Tue 17 Sep 2024 11:02:00 PM UTC. 2024-09-17T23:02:40.578 INFO:teuthology.orchestra.run.smithi006.stdout:(13/37): python3-markupsafe-1.1.1-12.el9.x86_64 321 kB/s | 35 kB 00:00 2024-09-17T23:02:40.636 INFO:teuthology.orchestra.run.smithi006.stdout:(14/37): python3-mako-1.1.4-6.el9.noarch.rpm 824 kB/s | 172 kB 00:00 2024-09-17T23:02:40.662 INFO:teuthology.orchestra.run.smithi006.stdout:(15/37): lua-5.4.4-4.el9.x86_64.rpm 726 kB/s | 188 kB 00:00 2024-09-17T23:02:40.687 INFO:teuthology.orchestra.run.smithi006.stdout:(16/37): python3-packaging-20.9-5.el9.noarch.rp 706 kB/s | 77 kB 00:00 2024-09-17T23:02:40.712 INFO:teuthology.orchestra.run.smithi006.stdout:(17/37): python3-pytz-2021.1-5.el9.noarch.rpm 673 kB/s | 51 kB 00:00 2024-09-17T23:02:40.720 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-09-17T23:02:40.723 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-17T23:02:40.723 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repository Size 2024-09-17T23:02:40.723 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-17T23:02:40.723 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-09-17T23:02:40.723 INFO:teuthology.orchestra.run.smithi038.stdout: ceph x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 6.5 k 2024-09-17T23:02:40.723 INFO:teuthology.orchestra.run.smithi038.stdout:Installing dependencies: 2024-09-17T23:02:40.723 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mds x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 2.4 M 2024-09-17T23:02:40.723 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 1.5 M 2024-09-17T23:02:40.724 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-modules-core noarch 2:19.1.1-330.gc2ddf972.el9 ceph-noarch 249 k 2024-09-17T23:02:40.724 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mon x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 4.7 M 2024-09-17T23:02:40.724 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-osd x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 17 M 2024-09-17T23:02:40.724 INFO:teuthology.orchestra.run.smithi038.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-17T23:02:40.724 INFO:teuthology.orchestra.run.smithi038.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-17T23:02:40.724 INFO:teuthology.orchestra.run.smithi038.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-17T23:02:40.724 INFO:teuthology.orchestra.run.smithi038.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-17T23:02:40.724 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-17T23:02:40.724 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-17T23:02:40.724 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-17T23:02:40.724 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-17T23:02:40.724 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-17T23:02:40.724 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-17T23:02:40.725 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-17T23:02:40.725 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-17T23:02:40.725 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-17T23:02:40.725 INFO:teuthology.orchestra.run.smithi038.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-17T23:02:40.725 INFO:teuthology.orchestra.run.smithi038.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-17T23:02:40.725 INFO:teuthology.orchestra.run.smithi038.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-17T23:02:40.725 INFO:teuthology.orchestra.run.smithi038.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-17T23:02:40.725 INFO:teuthology.orchestra.run.smithi038.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-17T23:02:40.725 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-17T23:02:40.725 INFO:teuthology.orchestra.run.smithi038.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-17T23:02:40.725 INFO:teuthology.orchestra.run.smithi038.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-17T23:02:40.725 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-17T23:02:40.725 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-17T23:02:40.725 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-17T23:02:40.726 INFO:teuthology.orchestra.run.smithi038.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-17T23:02:40.726 INFO:teuthology.orchestra.run.smithi038.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-17T23:02:40.726 INFO:teuthology.orchestra.run.smithi038.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-17T23:02:40.726 INFO:teuthology.orchestra.run.smithi038.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-17T23:02:40.726 INFO:teuthology.orchestra.run.smithi038.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-17T23:02:40.726 INFO:teuthology.orchestra.run.smithi038.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-17T23:02:40.726 INFO:teuthology.orchestra.run.smithi038.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-17T23:02:40.726 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:02:40.726 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-09-17T23:02:40.726 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-17T23:02:40.726 INFO:teuthology.orchestra.run.smithi038.stdout:Install 37 Packages 2024-09-17T23:02:40.726 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:02:40.727 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 30 M 2024-09-17T23:02:40.727 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 108 M 2024-09-17T23:02:40.727 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-09-17T23:02:40.738 INFO:teuthology.orchestra.run.smithi006.stdout:(18/37): python3-toml-0.10.2-6.el9.noarch.rpm 551 kB/s | 42 kB 00:00 2024-09-17T23:02:40.796 INFO:teuthology.orchestra.run.smithi006.stdout:(19/37): lua-devel-5.4.4-4.el9.x86_64.rpm 205 kB/s | 22 kB 00:00 2024-09-17T23:02:40.855 INFO:teuthology.orchestra.run.smithi006.stdout:(20/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 372 kB/s | 43 kB 00:00 2024-09-17T23:02:40.880 INFO:teuthology.orchestra.run.smithi006.stdout:(21/37): luarocks-3.9.2-5.el9.noarch.rpm 904 kB/s | 151 kB 00:00 2024-09-17T23:02:40.910 INFO:teuthology.orchestra.run.smithi006.stdout:(22/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 428 kB/s | 11 kB 00:00 2024-09-17T23:02:40.931 INFO:teuthology.orchestra.run.smithi006.stdout:(23/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.4 MB/s | 172 kB 00:00 2024-09-17T23:02:40.956 INFO:teuthology.orchestra.run.smithi006.stdout:(24/37): python3-jaraco-classes-3.2.1-5.el9.noa 353 kB/s | 18 kB 00:00 2024-09-17T23:02:40.981 INFO:teuthology.orchestra.run.smithi006.stdout:(25/37): python3-jaraco-collections-3.0.0-8.el9 462 kB/s | 23 kB 00:00 2024-09-17T23:02:41.015 INFO:teuthology.orchestra.run.smithi006.stdout:(26/37): python3-jaraco-functools-3.5.0-2.el9.n 332 kB/s | 19 kB 00:00 2024-09-17T23:02:41.040 INFO:teuthology.orchestra.run.smithi006.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 337 kB/s | 20 kB 00:00 2024-09-17T23:02:41.066 INFO:teuthology.orchestra.run.smithi006.stdout:(28/37): python3-cherrypy-18.6.1-2.el9.noarch.r 1.7 MB/s | 358 kB 00:00 2024-09-17T23:02:41.091 INFO:teuthology.orchestra.run.smithi006.stdout:(29/37): python3-logutils-0.3.5-21.el9.noarch.r 613 kB/s | 46 kB 00:00 2024-09-17T23:02:41.108 INFO:teuthology.orchestra.run.smithi006.stdout:(30/37): python3-more-itertools-8.12.0-2.el9.no 1.1 MB/s | 79 kB 00:00 2024-09-17T23:02:41.133 INFO:teuthology.orchestra.run.smithi006.stdout:(31/37): python3-portend-3.1.0-2.el9.noarch.rpm 395 kB/s | 16 kB 00:00 2024-09-17T23:02:41.158 INFO:teuthology.orchestra.run.smithi006.stdout:(32/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.8 MB/s | 90 kB 00:00 2024-09-17T23:02:41.183 INFO:teuthology.orchestra.run.smithi006.stdout:(33/37): python3-tempora-5.0.0-2.el9.noarch.rpm 716 kB/s | 36 kB 00:00 2024-09-17T23:02:41.209 INFO:teuthology.orchestra.run.smithi006.stdout:(34/37): python3-pecan-1.4.2-3.el9.noarch.rpm 2.3 MB/s | 272 kB 00:00 2024-09-17T23:02:41.234 INFO:teuthology.orchestra.run.smithi006.stdout:(35/37): python3-webob-1.8.8-2.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-09-17T23:02:41.259 INFO:teuthology.orchestra.run.smithi006.stdout:(36/37): python3-zc-lockfile-2.0-10.el9.noarch. 398 kB/s | 20 kB 00:00 2024-09-17T23:02:41.309 INFO:teuthology.orchestra.run.smithi006.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.3 MB/s | 427 kB 00:00 2024-09-17T23:02:41.314 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-09-17T23:02:41.314 INFO:teuthology.orchestra.run.smithi006.stdout:Total 13 MB/s | 30 MB 00:02 2024-09-17T23:02:41.472 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-09-17T23:02:41.517 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-09-17T23:02:41.517 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-09-17T23:02:41.590 INFO:teuthology.orchestra.run.smithi038.stdout:(1/37): ceph-19.1.1-330.gc2ddf972.el9.x86_64.rp 50 kB/s | 6.5 kB 00:00 2024-09-17T23:02:41.824 INFO:teuthology.orchestra.run.smithi038.stdout:(2/37): ceph-mds-19.1.1-330.gc2ddf972.el9.x86_6 6.5 MB/s | 2.4 MB 00:00 2024-09-17T23:02:41.866 INFO:teuthology.orchestra.run.smithi038.stdout:(3/37): ceph-mgr-19.1.1-330.gc2ddf972.el9.x86_6 3.7 MB/s | 1.5 MB 00:00 2024-09-17T23:02:41.950 INFO:teuthology.orchestra.run.smithi038.stdout:(4/37): ceph-mgr-modules-core-19.1.1-330.gc2ddf 2.9 MB/s | 249 kB 00:00 2024-09-17T23:02:42.007 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-09-17T23:02:42.007 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-09-17T23:02:42.016 INFO:teuthology.orchestra.run.smithi038.stdout:(5/37): ceph-mon-19.1.1-330.gc2ddf972.el9.x86_6 11 MB/s | 4.7 MB 00:00 2024-09-17T23:02:42.133 INFO:teuthology.orchestra.run.smithi038.stdout:(6/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.3 MB/s | 253 kB 00:00 2024-09-17T23:02:42.175 INFO:teuthology.orchestra.run.smithi038.stdout:(7/37): python3-ply-3.11-14.el9.noarch.rpm 2.5 MB/s | 106 kB 00:00 2024-09-17T23:02:42.213 INFO:teuthology.orchestra.run.smithi038.stdout:(8/37): python3-cryptography-36.0.1-4.el9.x86_6 6.3 MB/s | 1.2 MB 00:00 2024-09-17T23:02:42.238 INFO:teuthology.orchestra.run.smithi038.stdout:(9/37): python3-pycparser-2.20-6.el9.noarch.rpm 2.1 MB/s | 135 kB 00:00 2024-09-17T23:02:42.263 INFO:teuthology.orchestra.run.smithi038.stdout:(10/37): python3-requests-2.25.1-8.el9.noarch.r 2.4 MB/s | 125 kB 00:00 2024-09-17T23:02:42.438 INFO:teuthology.orchestra.run.smithi038.stdout:(11/37): ceph-osd-19.1.1-330.gc2ddf972.el9.x86_ 27 MB/s | 17 MB 00:00 2024-09-17T23:02:42.464 INFO:teuthology.orchestra.run.smithi038.stdout:(12/37): python3-urllib3-1.26.5-6.el9.noarch.rp 952 kB/s | 215 kB 00:00 2024-09-17T23:02:42.572 INFO:teuthology.orchestra.run.smithi038.stdout:(13/37): python3-markupsafe-1.1.1-12.el9.x86_64 321 kB/s | 35 kB 00:00 2024-09-17T23:02:42.606 INFO:teuthology.orchestra.run.smithi038.stdout:(14/37): lua-5.4.4-4.el9.x86_64.rpm 551 kB/s | 188 kB 00:00 2024-09-17T23:02:42.656 INFO:teuthology.orchestra.run.smithi038.stdout:(15/37): python3-packaging-20.9-5.el9.noarch.rp 922 kB/s | 77 kB 00:00 2024-09-17T23:02:42.681 INFO:teuthology.orchestra.run.smithi038.stdout:(16/37): python3-mako-1.1.4-6.el9.noarch.rpm 710 kB/s | 172 kB 00:00 2024-09-17T23:02:42.698 INFO:teuthology.orchestra.run.smithi038.stdout:(17/37): python3-pytz-2021.1-5.el9.noarch.rpm 552 kB/s | 51 kB 00:00 2024-09-17T23:02:42.715 INFO:teuthology.orchestra.run.smithi038.stdout:(18/37): python3-toml-0.10.2-6.el9.noarch.rpm 709 kB/s | 42 kB 00:00 2024-09-17T23:02:42.765 INFO:teuthology.orchestra.run.smithi038.stdout:(19/37): luarocks-3.9.2-5.el9.noarch.rpm 2.2 MB/s | 151 kB 00:00 2024-09-17T23:02:42.782 INFO:teuthology.orchestra.run.smithi038.stdout:(20/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 650 kB/s | 43 kB 00:00 2024-09-17T23:02:42.824 INFO:teuthology.orchestra.run.smithi038.stdout:(21/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.9 MB/s | 172 kB 00:00 2024-09-17T23:02:42.850 INFO:teuthology.orchestra.run.smithi038.stdout:(22/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 426 kB/s | 11 kB 00:00 2024-09-17T23:02:42.891 INFO:teuthology.orchestra.run.smithi038.stdout:(23/37): python3-cherrypy-18.6.1-2.el9.noarch.r 3.2 MB/s | 358 kB 00:00 2024-09-17T23:02:42.917 INFO:teuthology.orchestra.run.smithi038.stdout:(24/37): python3-jaraco-classes-3.2.1-5.el9.noa 264 kB/s | 18 kB 00:00 2024-09-17T23:02:42.942 INFO:teuthology.orchestra.run.smithi038.stdout:(25/37): python3-jaraco-collections-3.0.0-8.el9 461 kB/s | 23 kB 00:00 2024-09-17T23:02:42.968 INFO:teuthology.orchestra.run.smithi038.stdout:(26/37): python3-jaraco-functools-3.5.0-2.el9.n 387 kB/s | 19 kB 00:00 2024-09-17T23:02:42.993 INFO:teuthology.orchestra.run.smithi038.stdout:(27/37): lua-devel-5.4.4-4.el9.x86_64.rpm 72 kB/s | 22 kB 00:00 2024-09-17T23:02:43.018 INFO:teuthology.orchestra.run.smithi038.stdout:(28/37): python3-jaraco-text-3.2.0-6.el9.noarch 260 kB/s | 20 kB 00:00 2024-09-17T23:02:43.051 INFO:teuthology.orchestra.run.smithi038.stdout:(29/37): python3-logutils-0.3.5-21.el9.noarch.r 553 kB/s | 46 kB 00:00 2024-09-17T23:02:43.077 INFO:teuthology.orchestra.run.smithi038.stdout:(30/37): python3-portend-3.1.0-2.el9.noarch.rpm 708 kB/s | 16 kB 00:00 2024-09-17T23:02:43.110 INFO:teuthology.orchestra.run.smithi038.stdout:(31/37): python3-pecan-1.4.2-3.el9.noarch.rpm 3.1 MB/s | 272 kB 00:00 2024-09-17T23:02:43.136 INFO:teuthology.orchestra.run.smithi038.stdout:(32/37): python3-more-itertools-8.12.0-2.el9.no 551 kB/s | 79 kB 00:00 2024-09-17T23:02:43.161 INFO:teuthology.orchestra.run.smithi038.stdout:(33/37): python3-tempora-5.0.0-2.el9.noarch.rpm 709 kB/s | 36 kB 00:00 2024-09-17T23:02:43.186 INFO:teuthology.orchestra.run.smithi038.stdout:(34/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 823 kB/s | 90 kB 00:00 2024-09-17T23:02:43.212 INFO:teuthology.orchestra.run.smithi038.stdout:(35/37): python3-webob-1.8.8-2.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-09-17T23:02:43.237 INFO:teuthology.orchestra.run.smithi038.stdout:(36/37): python3-zc-lockfile-2.0-10.el9.noarch. 397 kB/s | 20 kB 00:00 2024-09-17T23:02:43.262 INFO:teuthology.orchestra.run.smithi038.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.2 MB/s | 427 kB 00:00 2024-09-17T23:02:43.266 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-09-17T23:02:43.266 INFO:teuthology.orchestra.run.smithi038.stdout:Total 12 MB/s | 30 MB 00:02 2024-09-17T23:02:43.364 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-09-17T23:02:43.414 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-09-17T23:02:43.429 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-17T23:02:43.459 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-09-17T23:02:43.460 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-09-17T23:02:43.480 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-17T23:02:43.526 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-17T23:02:43.607 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-17T23:02:43.648 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-17T23:02:43.681 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-17T23:02:43.777 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-17T23:02:43.863 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-17T23:02:43.909 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-17T23:02:43.943 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-17T23:02:43.948 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-09-17T23:02:43.948 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-09-17T23:02:43.990 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-17T23:02:44.028 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-17T23:02:44.058 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-17T23:02:44.099 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-17T23:02:44.142 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-17T23:02:44.176 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-17T23:02:44.226 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-17T23:02:44.265 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-17T23:02:44.351 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-17T23:02:44.503 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-17T23:02:44.594 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-17T23:02:44.652 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-17T23:02:44.690 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-17T23:02:44.749 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-17T23:02:44.812 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-17T23:02:45.021 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-17T23:02:45.077 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-17T23:02:45.159 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-17T23:02:45.248 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-17T23:02:45.285 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-17T23:02:45.353 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-09-17T23:02:45.415 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-17T23:02:45.417 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-17T23:02:45.472 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-17T23:02:45.501 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-17T23:02:45.577 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-17T23:02:45.588 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mgr-modules-core-2:19.1.1-330.gc2ddf972.el9.n 32/37 2024-09-17T23:02:45.617 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-17T23:02:45.619 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mgr-2:19.1.1-330.gc2ddf972.el9.x86_64 33/37 2024-09-17T23:02:45.646 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-mgr-2:19.1.1-330.gc2ddf972.el9.x86_64 33/37 2024-09-17T23:02:45.647 INFO:teuthology.orchestra.run.smithi006.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T23:02:45.647 INFO:teuthology.orchestra.run.smithi006.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-17T23:02:45.647 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-17T23:02:45.647 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-17T23:02:45.647 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:02:45.658 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-17T23:02:45.755 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-17T23:02:45.832 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-17T23:02:45.887 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-17T23:02:45.929 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-17T23:02:45.959 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-17T23:02:45.994 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-17T23:02:46.027 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-17T23:02:46.076 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-17T23:02:46.121 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-17T23:02:46.163 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-17T23:02:46.212 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-17T23:02:46.252 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-17T23:02:46.329 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-17T23:02:46.481 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-17T23:02:46.572 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-17T23:02:46.634 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-17T23:02:46.679 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-17T23:02:46.737 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-17T23:02:46.801 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-17T23:02:46.997 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-17T23:02:47.049 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-17T23:02:47.131 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-17T23:02:47.178 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-osd-2:19.1.1-330.gc2ddf972.el9.x86_64 34/37 2024-09-17T23:02:47.207 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-osd-2:19.1.1-330.gc2ddf972.el9.x86_64 34/37 2024-09-17T23:02:47.207 INFO:teuthology.orchestra.run.smithi006.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T23:02:47.207 INFO:teuthology.orchestra.run.smithi006.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-17T23:02:47.207 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-17T23:02:47.207 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-17T23:02:47.207 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:02:47.228 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-17T23:02:47.270 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-17T23:02:47.398 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-17T23:02:47.577 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mgr-modules-core-2:19.1.1-330.gc2ddf972.el9.n 32/37 2024-09-17T23:02:47.633 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mgr-2:19.1.1-330.gc2ddf972.el9.x86_64 33/37 2024-09-17T23:02:47.661 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mgr-2:19.1.1-330.gc2ddf972.el9.x86_64 33/37 2024-09-17T23:02:47.661 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T23:02:47.661 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-17T23:02:47.661 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-17T23:02:47.661 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-17T23:02:47.661 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:02:47.662 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mon-2:19.1.1-330.gc2ddf972.el9.x86_64 35/37 2024-09-17T23:02:47.687 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-mon-2:19.1.1-330.gc2ddf972.el9.x86_64 35/37 2024-09-17T23:02:47.687 INFO:teuthology.orchestra.run.smithi006.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T23:02:47.687 INFO:teuthology.orchestra.run.smithi006.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-17T23:02:47.687 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-17T23:02:47.688 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-17T23:02:47.688 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:02:47.921 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mds-2:19.1.1-330.gc2ddf972.el9.x86_64 36/37 2024-09-17T23:02:47.946 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-mds-2:19.1.1-330.gc2ddf972.el9.x86_64 36/37 2024-09-17T23:02:47.946 INFO:teuthology.orchestra.run.smithi006.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T23:02:47.946 INFO:teuthology.orchestra.run.smithi006.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-17T23:02:47.946 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-17T23:02:47.946 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-17T23:02:47.946 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:02:48.022 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-2:19.1.1-330.gc2ddf972.el9.x86_64 37/37 2024-09-17T23:02:49.192 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-osd-2:19.1.1-330.gc2ddf972.el9.x86_64 34/37 2024-09-17T23:02:49.220 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-osd-2:19.1.1-330.gc2ddf972.el9.x86_64 34/37 2024-09-17T23:02:49.221 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T23:02:49.221 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-17T23:02:49.221 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-17T23:02:49.221 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-17T23:02:49.221 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:02:49.693 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mon-2:19.1.1-330.gc2ddf972.el9.x86_64 35/37 2024-09-17T23:02:49.718 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mon-2:19.1.1-330.gc2ddf972.el9.x86_64 35/37 2024-09-17T23:02:49.718 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T23:02:49.718 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-17T23:02:49.718 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-17T23:02:49.718 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-17T23:02:49.718 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:02:49.857 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-2:19.1.1-330.gc2ddf972.el9.x86_64 37/37 2024-09-17T23:02:49.857 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-2:19.1.1-330.gc2ddf972.el9.x86_64 1/37 2024-09-17T23:02:49.857 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mds-2:19.1.1-330.gc2ddf972.el9.x86_64 2/37 2024-09-17T23:02:49.857 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mgr-2:19.1.1-330.gc2ddf972.el9.x86_64 3/37 2024-09-17T23:02:49.857 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mon-2:19.1.1-330.gc2ddf972.el9.x86_64 4/37 2024-09-17T23:02:49.857 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-osd-2:19.1.1-330.gc2ddf972.el9.x86_64 5/37 2024-09-17T23:02:49.857 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mgr-modules-core-2:19.1.1-330.gc2ddf972.el9.n 6/37 2024-09-17T23:02:49.857 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-17T23:02:49.858 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-17T23:02:49.858 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-17T23:02:49.858 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-17T23:02:49.858 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-17T23:02:49.858 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-17T23:02:49.858 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-17T23:02:49.858 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-17T23:02:49.858 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-17T23:02:49.858 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-17T23:02:49.858 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-17T23:02:49.858 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-17T23:02:49.858 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-17T23:02:49.858 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-17T23:02:49.858 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-17T23:02:49.859 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-17T23:02:49.859 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-17T23:02:49.859 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-17T23:02:49.859 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-17T23:02:49.859 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-17T23:02:49.859 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-17T23:02:49.859 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-17T23:02:49.859 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-17T23:02:49.859 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-17T23:02:49.860 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-17T23:02:49.860 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-17T23:02:49.860 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-17T23:02:49.860 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-17T23:02:49.860 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-17T23:02:49.860 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-17T23:02:49.960 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mds-2:19.1.1-330.gc2ddf972.el9.x86_64 36/37 2024-09-17T23:02:49.985 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mds-2:19.1.1-330.gc2ddf972.el9.x86_64 36/37 2024-09-17T23:02:49.985 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T23:02:49.985 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-17T23:02:49.985 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-17T23:02:49.985 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-17T23:02:49.986 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:02:50.052 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-2:19.1.1-330.gc2ddf972.el9.x86_64 37/37 2024-09-17T23:02:51.079 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-17T23:02:51.079 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:02:51.079 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-09-17T23:02:51.079 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:02:51.079 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mds-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:02:51.079 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:02:51.079 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-modules-core-2:19.1.1-330.gc2ddf972.el9.noarch 2024-09-17T23:02:51.079 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mon-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:02:51.079 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-osd-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:02:51.080 INFO:teuthology.orchestra.run.smithi006.stdout: lua-5.4.4-4.el9.x86_64 2024-09-17T23:02:51.080 INFO:teuthology.orchestra.run.smithi006.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-17T23:02:51.080 INFO:teuthology.orchestra.run.smithi006.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-17T23:02:51.080 INFO:teuthology.orchestra.run.smithi006.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-17T23:02:51.080 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-17T23:02:51.080 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-17T23:02:51.080 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-17T23:02:51.080 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-17T23:02:51.080 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-17T23:02:51.080 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-17T23:02:51.080 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-17T23:02:51.080 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-17T23:02:51.080 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-17T23:02:51.080 INFO:teuthology.orchestra.run.smithi006.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-17T23:02:51.080 INFO:teuthology.orchestra.run.smithi006.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-17T23:02:51.081 INFO:teuthology.orchestra.run.smithi006.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-17T23:02:51.081 INFO:teuthology.orchestra.run.smithi006.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-17T23:02:51.081 INFO:teuthology.orchestra.run.smithi006.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-17T23:02:51.081 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-17T23:02:51.081 INFO:teuthology.orchestra.run.smithi006.stdout: python3-ply-3.11-14.el9.noarch 2024-09-17T23:02:51.081 INFO:teuthology.orchestra.run.smithi006.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-17T23:02:51.081 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-17T23:02:51.081 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-17T23:02:51.081 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-17T23:02:51.081 INFO:teuthology.orchestra.run.smithi006.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-17T23:02:51.081 INFO:teuthology.orchestra.run.smithi006.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-17T23:02:51.081 INFO:teuthology.orchestra.run.smithi006.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-17T23:02:51.081 INFO:teuthology.orchestra.run.smithi006.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-17T23:02:51.081 INFO:teuthology.orchestra.run.smithi006.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-17T23:02:51.081 INFO:teuthology.orchestra.run.smithi006.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-17T23:02:51.082 INFO:teuthology.orchestra.run.smithi006.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-17T23:02:51.082 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:02:51.082 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-09-17T23:02:51.457 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install ceph-base 2024-09-17T23:02:51.812 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-2:19.1.1-330.gc2ddf972.el9.x86_64 37/37 2024-09-17T23:02:51.812 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-2:19.1.1-330.gc2ddf972.el9.x86_64 1/37 2024-09-17T23:02:51.812 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mds-2:19.1.1-330.gc2ddf972.el9.x86_64 2/37 2024-09-17T23:02:51.812 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mgr-2:19.1.1-330.gc2ddf972.el9.x86_64 3/37 2024-09-17T23:02:51.812 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mon-2:19.1.1-330.gc2ddf972.el9.x86_64 4/37 2024-09-17T23:02:51.812 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-osd-2:19.1.1-330.gc2ddf972.el9.x86_64 5/37 2024-09-17T23:02:51.812 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mgr-modules-core-2:19.1.1-330.gc2ddf972.el9.n 6/37 2024-09-17T23:02:51.812 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-17T23:02:51.812 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-17T23:02:51.812 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-17T23:02:51.812 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-17T23:02:51.813 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-17T23:02:51.813 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-17T23:02:51.813 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-17T23:02:51.813 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-17T23:02:51.813 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-17T23:02:51.813 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-17T23:02:51.813 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-17T23:02:51.813 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-17T23:02:51.813 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-17T23:02:51.813 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-17T23:02:51.813 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-17T23:02:51.813 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-17T23:02:51.813 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-17T23:02:51.814 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-17T23:02:51.814 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-17T23:02:51.814 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-17T23:02:51.814 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-17T23:02:51.814 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-17T23:02:51.814 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-17T23:02:51.814 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-17T23:02:51.814 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-17T23:02:51.814 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-17T23:02:51.814 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-17T23:02:51.814 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-17T23:02:51.814 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-17T23:02:51.814 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-17T23:02:52.073 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:00:55 ago on Tue 17 Sep 2024 11:01:57 PM UTC. 2024-09-17T23:02:52.165 INFO:teuthology.orchestra.run.smithi006.stdout:Package ceph-base-2:19.1.1-330.gc2ddf972.el9.x86_64 is already installed. 2024-09-17T23:02:52.210 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-09-17T23:02:52.211 INFO:teuthology.orchestra.run.smithi006.stdout:Nothing to do. 2024-09-17T23:02:52.212 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-09-17T23:02:52.275 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install cephadm 2024-09-17T23:02:52.876 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:00:55 ago on Tue 17 Sep 2024 11:01:57 PM UTC. 2024-09-17T23:02:52.915 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-17T23:02:52.916 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:02:52.916 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-09-17T23:02:52.916 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:02:52.916 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mds-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:02:52.916 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:02:52.916 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-modules-core-2:19.1.1-330.gc2ddf972.el9.noarch 2024-09-17T23:02:52.916 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mon-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:02:52.916 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-osd-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:02:52.916 INFO:teuthology.orchestra.run.smithi038.stdout: lua-5.4.4-4.el9.x86_64 2024-09-17T23:02:52.916 INFO:teuthology.orchestra.run.smithi038.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-17T23:02:52.916 INFO:teuthology.orchestra.run.smithi038.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-17T23:02:52.916 INFO:teuthology.orchestra.run.smithi038.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-17T23:02:52.917 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-17T23:02:52.917 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-17T23:02:52.917 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-17T23:02:52.917 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-17T23:02:52.917 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-17T23:02:52.917 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-17T23:02:52.917 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-17T23:02:52.917 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-17T23:02:52.917 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-17T23:02:52.917 INFO:teuthology.orchestra.run.smithi038.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-17T23:02:52.917 INFO:teuthology.orchestra.run.smithi038.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-17T23:02:52.917 INFO:teuthology.orchestra.run.smithi038.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-17T23:02:52.917 INFO:teuthology.orchestra.run.smithi038.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-17T23:02:52.917 INFO:teuthology.orchestra.run.smithi038.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-17T23:02:52.918 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-17T23:02:52.918 INFO:teuthology.orchestra.run.smithi038.stdout: python3-ply-3.11-14.el9.noarch 2024-09-17T23:02:52.918 INFO:teuthology.orchestra.run.smithi038.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-17T23:02:52.918 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-17T23:02:52.918 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-17T23:02:52.918 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-17T23:02:52.918 INFO:teuthology.orchestra.run.smithi038.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-17T23:02:52.918 INFO:teuthology.orchestra.run.smithi038.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-17T23:02:52.918 INFO:teuthology.orchestra.run.smithi038.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-17T23:02:52.918 INFO:teuthology.orchestra.run.smithi038.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-17T23:02:52.918 INFO:teuthology.orchestra.run.smithi038.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-17T23:02:52.918 INFO:teuthology.orchestra.run.smithi038.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-17T23:02:52.918 INFO:teuthology.orchestra.run.smithi038.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-17T23:02:52.918 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:02:52.919 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-09-17T23:02:53.012 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-09-17T23:02:53.013 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-09-17T23:02:53.013 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repository Size 2024-09-17T23:02:53.013 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-09-17T23:02:53.013 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-09-17T23:02:53.013 INFO:teuthology.orchestra.run.smithi006.stdout: cephadm noarch 2:19.1.1-330.gc2ddf972.el9 ceph-noarch 765 k 2024-09-17T23:02:53.013 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:02:53.013 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-09-17T23:02:53.013 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-09-17T23:02:53.013 INFO:teuthology.orchestra.run.smithi006.stdout:Install 1 Package 2024-09-17T23:02:53.013 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:02:53.014 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 765 k 2024-09-17T23:02:53.014 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 771 k 2024-09-17T23:02:53.014 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-09-17T23:02:53.279 INFO:teuthology.orchestra.run.smithi006.stdout:cephadm-19.1.1-330.gc2ddf972.el9.noarch.rpm 2.8 MB/s | 765 kB 00:00 2024-09-17T23:02:53.279 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-09-17T23:02:53.279 INFO:teuthology.orchestra.run.smithi006.stdout:Total 2.8 MB/s | 765 kB 00:00 2024-09-17T23:02:53.280 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-09-17T23:02:53.285 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-09-17T23:02:53.285 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-09-17T23:02:53.291 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-09-17T23:02:53.292 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-09-17T23:02:53.294 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-base 2024-09-17T23:02:53.363 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-09-17T23:02:53.868 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: cephadm-2:19.1.1-330.gc2ddf972.el9.noarch 1/1 2024-09-17T23:02:53.904 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:00:53 ago on Tue 17 Sep 2024 11:02:00 PM UTC. 2024-09-17T23:02:53.969 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : cephadm-2:19.1.1-330.gc2ddf972.el9.noarch 1/1 2024-09-17T23:02:53.995 INFO:teuthology.orchestra.run.smithi038.stdout:Package ceph-base-2:19.1.1-330.gc2ddf972.el9.x86_64 is already installed. 2024-09-17T23:02:54.039 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-09-17T23:02:54.040 INFO:teuthology.orchestra.run.smithi038.stdout:Nothing to do. 2024-09-17T23:02:54.040 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-09-17T23:02:54.105 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install cephadm 2024-09-17T23:02:54.312 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: cephadm-2:19.1.1-330.gc2ddf972.el9.noarch 1/1 2024-09-17T23:02:54.560 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : cephadm-2:19.1.1-330.gc2ddf972.el9.noarch 1/1 2024-09-17T23:02:54.560 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:02:54.560 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-09-17T23:02:54.560 INFO:teuthology.orchestra.run.smithi006.stdout: cephadm-2:19.1.1-330.gc2ddf972.el9.noarch 2024-09-17T23:02:54.560 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:02:54.560 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-09-17T23:02:54.711 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:00:54 ago on Tue 17 Sep 2024 11:02:00 PM UTC. 2024-09-17T23:02:54.750 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install ceph-immutable-object-cache 2024-09-17T23:02:54.849 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-09-17T23:02:54.849 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-17T23:02:54.850 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repository Size 2024-09-17T23:02:54.850 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-17T23:02:54.850 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-09-17T23:02:54.850 INFO:teuthology.orchestra.run.smithi038.stdout: cephadm noarch 2:19.1.1-330.gc2ddf972.el9 ceph-noarch 765 k 2024-09-17T23:02:54.850 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:02:54.850 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-09-17T23:02:54.850 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-17T23:02:54.850 INFO:teuthology.orchestra.run.smithi038.stdout:Install 1 Package 2024-09-17T23:02:54.850 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:02:54.851 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 765 k 2024-09-17T23:02:54.851 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 771 k 2024-09-17T23:02:54.851 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-09-17T23:02:55.111 INFO:teuthology.orchestra.run.smithi038.stdout:cephadm-19.1.1-330.gc2ddf972.el9.noarch.rpm 2.9 MB/s | 765 kB 00:00 2024-09-17T23:02:55.111 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-09-17T23:02:55.112 INFO:teuthology.orchestra.run.smithi038.stdout:Total 2.9 MB/s | 765 kB 00:00 2024-09-17T23:02:55.112 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-09-17T23:02:55.117 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-09-17T23:02:55.117 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-09-17T23:02:55.123 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-09-17T23:02:55.124 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-09-17T23:02:55.202 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-09-17T23:02:55.356 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:00:58 ago on Tue 17 Sep 2024 11:01:57 PM UTC. 2024-09-17T23:02:55.494 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-09-17T23:02:55.495 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-09-17T23:02:55.495 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repo Size 2024-09-17T23:02:55.495 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-09-17T23:02:55.495 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-09-17T23:02:55.495 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-immutable-object-cache x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 145 k 2024-09-17T23:02:55.495 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:02:55.495 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-09-17T23:02:55.495 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-09-17T23:02:55.495 INFO:teuthology.orchestra.run.smithi006.stdout:Install 1 Package 2024-09-17T23:02:55.495 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:02:55.495 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 145 k 2024-09-17T23:02:55.495 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 435 k 2024-09-17T23:02:55.496 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-09-17T23:02:55.694 INFO:teuthology.orchestra.run.smithi006.stdout:ceph-immutable-object-cache-19.1.1-330.gc2ddf97 734 kB/s | 145 kB 00:00 2024-09-17T23:02:55.695 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-09-17T23:02:55.695 INFO:teuthology.orchestra.run.smithi006.stdout:Total 729 kB/s | 145 kB 00:00 2024-09-17T23:02:55.695 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-09-17T23:02:55.700 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: cephadm-2:19.1.1-330.gc2ddf972.el9.noarch 1/1 2024-09-17T23:02:55.703 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-09-17T23:02:55.703 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-09-17T23:02:55.764 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-09-17T23:02:55.765 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-09-17T23:02:55.801 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : cephadm-2:19.1.1-330.gc2ddf972.el9.noarch 1/1 2024-09-17T23:02:55.916 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-09-17T23:02:55.987 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-immutable-object-cache-2:19.1.1-330.gc2ddf972.e 1/1 2024-09-17T23:02:56.014 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.1.1-330.gc2ddf972.e 1/1 2024-09-17T23:02:56.014 INFO:teuthology.orchestra.run.smithi006.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T23:02:56.014 INFO:teuthology.orchestra.run.smithi006.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-17T23:02:56.014 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:02:56.192 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: cephadm-2:19.1.1-330.gc2ddf972.el9.noarch 1/1 2024-09-17T23:02:56.454 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : cephadm-2:19.1.1-330.gc2ddf972.el9.noarch 1/1 2024-09-17T23:02:56.454 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:02:56.454 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-09-17T23:02:56.455 INFO:teuthology.orchestra.run.smithi038.stdout: cephadm-2:19.1.1-330.gc2ddf972.el9.noarch 2024-09-17T23:02:56.455 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:02:56.455 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-09-17T23:02:56.650 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-immutable-object-cache 2024-09-17T23:02:56.959 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-immutable-object-cache-2:19.1.1-330.gc2ddf972.e 1/1 2024-09-17T23:02:56.960 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:02:56.960 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-09-17T23:02:56.960 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-immutable-object-cache-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:02:56.960 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:02:56.960 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-09-17T23:02:57.233 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install ceph-mgr 2024-09-17T23:02:57.260 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:00:57 ago on Tue 17 Sep 2024 11:02:00 PM UTC. 2024-09-17T23:02:57.401 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-09-17T23:02:57.401 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-17T23:02:57.402 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repo Size 2024-09-17T23:02:57.402 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-17T23:02:57.402 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-09-17T23:02:57.402 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-immutable-object-cache x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 145 k 2024-09-17T23:02:57.402 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:02:57.402 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-09-17T23:02:57.402 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-17T23:02:57.402 INFO:teuthology.orchestra.run.smithi038.stdout:Install 1 Package 2024-09-17T23:02:57.402 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:02:57.402 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 145 k 2024-09-17T23:02:57.402 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 435 k 2024-09-17T23:02:57.402 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-09-17T23:02:57.617 INFO:teuthology.orchestra.run.smithi038.stdout:ceph-immutable-object-cache-19.1.1-330.gc2ddf97 679 kB/s | 145 kB 00:00 2024-09-17T23:02:57.617 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-09-17T23:02:57.617 INFO:teuthology.orchestra.run.smithi038.stdout:Total 674 kB/s | 145 kB 00:00 2024-09-17T23:02:57.617 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-09-17T23:02:57.626 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-09-17T23:02:57.626 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-09-17T23:02:57.686 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-09-17T23:02:57.686 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-09-17T23:02:57.835 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:01:00 ago on Tue 17 Sep 2024 11:01:57 PM UTC. 2024-09-17T23:02:57.844 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-09-17T23:02:57.918 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-immutable-object-cache-2:19.1.1-330.gc2ddf972.e 1/1 2024-09-17T23:02:57.928 INFO:teuthology.orchestra.run.smithi006.stdout:Package ceph-mgr-2:19.1.1-330.gc2ddf972.el9.x86_64 is already installed. 2024-09-17T23:02:57.944 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.1.1-330.gc2ddf972.e 1/1 2024-09-17T23:02:57.944 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T23:02:57.944 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-17T23:02:57.944 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:02:57.973 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-09-17T23:02:57.974 INFO:teuthology.orchestra.run.smithi006.stdout:Nothing to do. 2024-09-17T23:02:57.974 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-09-17T23:02:58.043 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install ceph-mgr-dashboard 2024-09-17T23:02:58.642 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:01:01 ago on Tue 17 Sep 2024 11:01:57 PM UTC. 2024-09-17T23:02:58.779 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-09-17T23:02:58.780 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-09-17T23:02:58.780 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repository Size 2024-09-17T23:02:58.780 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-09-17T23:02:58.781 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-09-17T23:02:58.781 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-dashboard noarch 2:19.1.1-330.gc2ddf972.el9 ceph-noarch 3.6 M 2024-09-17T23:02:58.781 INFO:teuthology.orchestra.run.smithi006.stdout:Installing dependencies: 2024-09-17T23:02:58.781 INFO:teuthology.orchestra.run.smithi006.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-17T23:02:58.781 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-grafana-dashboards noarch 2:19.1.1-330.gc2ddf972.el9 ceph-noarch 32 k 2024-09-17T23:02:58.781 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-prometheus-alerts noarch 2:19.1.1-330.gc2ddf972.el9 ceph-noarch 16 k 2024-09-17T23:02:58.781 INFO:teuthology.orchestra.run.smithi006.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-17T23:02:58.781 INFO:teuthology.orchestra.run.smithi006.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-17T23:02:58.781 INFO:teuthology.orchestra.run.smithi006.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-17T23:02:58.781 INFO:teuthology.orchestra.run.smithi006.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-17T23:02:58.781 INFO:teuthology.orchestra.run.smithi006.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-17T23:02:58.781 INFO:teuthology.orchestra.run.smithi006.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-17T23:02:58.782 INFO:teuthology.orchestra.run.smithi006.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-17T23:02:58.782 INFO:teuthology.orchestra.run.smithi006.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-17T23:02:58.782 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:02:58.782 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-09-17T23:02:58.782 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-09-17T23:02:58.782 INFO:teuthology.orchestra.run.smithi006.stdout:Install 12 Packages 2024-09-17T23:02:58.782 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:02:58.782 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 8.7 M 2024-09-17T23:02:58.783 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 104 M 2024-09-17T23:02:58.783 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-09-17T23:02:58.893 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-immutable-object-cache-2:19.1.1-330.gc2ddf972.e 1/1 2024-09-17T23:02:58.893 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:02:58.894 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-09-17T23:02:58.894 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-immutable-object-cache-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:02:58.894 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:02:58.894 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-09-17T23:02:59.149 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-mgr 2024-09-17T23:02:59.458 INFO:teuthology.orchestra.run.smithi006.stdout:(1/12): ceph-grafana-dashboards-19.1.1-330.gc2d 208 kB/s | 32 kB 00:00 2024-09-17T23:02:59.483 INFO:teuthology.orchestra.run.smithi006.stdout:(2/12): ceph-prometheus-alerts-19.1.1-330.gc2dd 92 kB/s | 16 kB 00:00 2024-09-17T23:02:59.625 INFO:teuthology.orchestra.run.smithi006.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.9 MB/s | 269 kB 00:00 2024-09-17T23:02:59.675 INFO:teuthology.orchestra.run.smithi006.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.7 MB/s | 1.0 MB 00:00 2024-09-17T23:02:59.742 INFO:teuthology.orchestra.run.smithi006.stdout:(5/12): ceph-mgr-dashboard-19.1.1-330.gc2ddf972 8.1 MB/s | 3.6 MB 00:00 2024-09-17T23:02:59.756 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:00:59 ago on Tue 17 Sep 2024 11:02:00 PM UTC. 2024-09-17T23:02:59.793 INFO:teuthology.orchestra.run.smithi006.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 407 kB/s | 19 kB 00:00 2024-09-17T23:02:59.843 INFO:teuthology.orchestra.run.smithi006.stdout:(7/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 3.2 MB/s | 548 kB 00:00 2024-09-17T23:02:59.847 INFO:teuthology.orchestra.run.smithi038.stdout:Package ceph-mgr-2:19.1.1-330.gc2ddf972.el9.x86_64 is already installed. 2024-09-17T23:02:59.877 INFO:teuthology.orchestra.run.smithi006.stdout:(8/12): protobuf-compiler-3.14.0-13.el9.x86_64. 3.4 MB/s | 863 kB 00:00 2024-09-17T23:02:59.891 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-09-17T23:02:59.893 INFO:teuthology.orchestra.run.smithi038.stdout:Nothing to do. 2024-09-17T23:02:59.893 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-09-17T23:02:59.902 INFO:teuthology.orchestra.run.smithi006.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 2.4 MB/s | 144 kB 00:00 2024-09-17T23:02:59.952 INFO:teuthology.orchestra.run.smithi006.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 410 kB/s | 31 kB 00:00 2024-09-17T23:02:59.957 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-mgr-dashboard 2024-09-17T23:02:59.977 INFO:teuthology.orchestra.run.smithi006.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.7 MB/s | 188 kB 00:00 2024-09-17T23:03:00.235 INFO:teuthology.orchestra.run.smithi006.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 4.6 MB/s | 2.0 MB 00:00 2024-09-17T23:03:00.240 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-09-17T23:03:00.241 INFO:teuthology.orchestra.run.smithi006.stdout:Total 5.9 MB/s | 8.7 MB 00:01 2024-09-17T23:03:00.293 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-09-17T23:03:00.313 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-09-17T23:03:00.314 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-09-17T23:03:00.528 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-09-17T23:03:00.528 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-09-17T23:03:00.559 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:00 ago on Tue 17 Sep 2024 11:02:00 PM UTC. 2024-09-17T23:03:00.697 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-09-17T23:03:00.698 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-17T23:03:00.698 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repository Size 2024-09-17T23:03:00.698 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-17T23:03:00.698 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-09-17T23:03:00.698 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-dashboard noarch 2:19.1.1-330.gc2ddf972.el9 ceph-noarch 3.6 M 2024-09-17T23:03:00.698 INFO:teuthology.orchestra.run.smithi038.stdout:Installing dependencies: 2024-09-17T23:03:00.698 INFO:teuthology.orchestra.run.smithi038.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-17T23:03:00.698 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-grafana-dashboards noarch 2:19.1.1-330.gc2ddf972.el9 ceph-noarch 32 k 2024-09-17T23:03:00.699 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-prometheus-alerts noarch 2:19.1.1-330.gc2ddf972.el9 ceph-noarch 16 k 2024-09-17T23:03:00.699 INFO:teuthology.orchestra.run.smithi038.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-17T23:03:00.699 INFO:teuthology.orchestra.run.smithi038.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-17T23:03:00.699 INFO:teuthology.orchestra.run.smithi038.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-17T23:03:00.699 INFO:teuthology.orchestra.run.smithi038.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-17T23:03:00.699 INFO:teuthology.orchestra.run.smithi038.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-17T23:03:00.699 INFO:teuthology.orchestra.run.smithi038.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-17T23:03:00.699 INFO:teuthology.orchestra.run.smithi038.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-17T23:03:00.699 INFO:teuthology.orchestra.run.smithi038.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-17T23:03:00.699 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:03:00.699 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-09-17T23:03:00.699 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-17T23:03:00.699 INFO:teuthology.orchestra.run.smithi038.stdout:Install 12 Packages 2024-09-17T23:03:00.699 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:03:00.700 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 8.7 M 2024-09-17T23:03:00.700 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 104 M 2024-09-17T23:03:00.700 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-09-17T23:03:01.082 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-09-17T23:03:01.154 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-17T23:03:01.189 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-17T23:03:01.236 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-17T23:03:01.273 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-17T23:03:01.370 INFO:teuthology.orchestra.run.smithi038.stdout:(1/12): ceph-grafana-dashboards-19.1.1-330.gc2d 221 kB/s | 32 kB 00:00 2024-09-17T23:03:01.383 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-17T23:03:01.387 INFO:teuthology.orchestra.run.smithi038.stdout:(2/12): ceph-prometheus-alerts-19.1.1-330.gc2dd 102 kB/s | 16 kB 00:00 2024-09-17T23:03:01.523 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-17T23:03:01.587 INFO:teuthology.orchestra.run.smithi038.stdout:(3/12): ceph-mgr-dashboard-19.1.1-330.gc2ddf972 9.9 MB/s | 3.6 MB 00:00 2024-09-17T23:03:01.613 INFO:teuthology.orchestra.run.smithi038.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.2 MB/s | 269 kB 00:00 2024-09-17T23:03:01.613 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-17T23:03:01.654 INFO:teuthology.orchestra.run.smithi038.stdout:(5/12): protobuf-3.14.0-13.el9.x86_64.rpm 3.6 MB/s | 1.0 MB 00:00 2024-09-17T23:03:01.663 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-17T23:03:01.690 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-17T23:03:01.722 INFO:teuthology.orchestra.run.smithi038.stdout:(6/12): protobuf-compiler-3.14.0-13.el9.x86_64. 6.3 MB/s | 863 kB 00:00 2024-09-17T23:03:01.722 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-prometheus-alerts-2:19.1.1-330.gc2ddf972.el9. 10/12 2024-09-17T23:03:01.738 INFO:teuthology.orchestra.run.smithi038.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 255 kB/s | 19 kB 00:00 2024-09-17T23:03:01.789 INFO:teuthology.orchestra.run.smithi038.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 3.3 MB/s | 144 kB 00:00 2024-09-17T23:03:01.814 INFO:teuthology.orchestra.run.smithi038.stdout:(9/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 2.7 MB/s | 548 kB 00:00 2024-09-17T23:03:01.831 INFO:teuthology.orchestra.run.smithi038.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 736 kB/s | 31 kB 00:00 2024-09-17T23:03:01.865 INFO:teuthology.orchestra.run.smithi038.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 3.7 MB/s | 188 kB 00:00 2024-09-17T23:03:02.115 INFO:teuthology.orchestra.run.smithi038.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 5.2 MB/s | 2.0 MB 00:00 2024-09-17T23:03:02.121 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-09-17T23:03:02.121 INFO:teuthology.orchestra.run.smithi038.stdout:Total 6.1 MB/s | 8.7 MB 00:01 2024-09-17T23:03:02.175 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-09-17T23:03:02.195 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-09-17T23:03:02.195 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-09-17T23:03:02.426 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-09-17T23:03:02.427 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-09-17T23:03:02.835 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-grafana-dashboards-2:19.1.1-330.gc2ddf972.el9 11/12 2024-09-17T23:03:02.872 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mgr-dashboard-2:19.1.1-330.gc2ddf972.el9.noar 12/12 2024-09-17T23:03:02.973 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-09-17T23:03:03.092 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-17T23:03:03.135 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-17T23:03:03.182 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-17T23:03:03.211 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-17T23:03:03.312 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-17T23:03:03.452 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-17T23:03:03.553 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-17T23:03:03.600 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-17T23:03:03.627 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-17T23:03:03.660 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-prometheus-alerts-2:19.1.1-330.gc2ddf972.el9. 10/12 2024-09-17T23:03:04.340 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.1.1-330.gc2ddf972.el9.noar 12/12 2024-09-17T23:03:04.340 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-grafana-dashboards-2:19.1.1-330.gc2ddf972.el9 1/12 2024-09-17T23:03:04.340 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mgr-dashboard-2:19.1.1-330.gc2ddf972.el9.noar 2/12 2024-09-17T23:03:04.340 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-prometheus-alerts-2:19.1.1-330.gc2ddf972.el9. 3/12 2024-09-17T23:03:04.340 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-17T23:03:04.340 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-17T23:03:04.340 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-17T23:03:04.340 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-17T23:03:04.340 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-17T23:03:04.340 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-17T23:03:04.340 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-17T23:03:04.341 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-17T23:03:04.667 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-17T23:03:04.667 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:03:04.667 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-09-17T23:03:04.668 INFO:teuthology.orchestra.run.smithi006.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-17T23:03:04.668 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-grafana-dashboards-2:19.1.1-330.gc2ddf972.el9.noarch 2024-09-17T23:03:04.668 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-dashboard-2:19.1.1-330.gc2ddf972.el9.noarch 2024-09-17T23:03:04.668 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-prometheus-alerts-2:19.1.1-330.gc2ddf972.el9.noarch 2024-09-17T23:03:04.668 INFO:teuthology.orchestra.run.smithi006.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-17T23:03:04.668 INFO:teuthology.orchestra.run.smithi006.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-17T23:03:04.668 INFO:teuthology.orchestra.run.smithi006.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-17T23:03:04.668 INFO:teuthology.orchestra.run.smithi006.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-17T23:03:04.668 INFO:teuthology.orchestra.run.smithi006.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-17T23:03:04.668 INFO:teuthology.orchestra.run.smithi006.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-17T23:03:04.668 INFO:teuthology.orchestra.run.smithi006.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-17T23:03:04.668 INFO:teuthology.orchestra.run.smithi006.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-17T23:03:04.668 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:03:04.669 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-09-17T23:03:04.773 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-grafana-dashboards-2:19.1.1-330.gc2ddf972.el9 11/12 2024-09-17T23:03:04.810 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mgr-dashboard-2:19.1.1-330.gc2ddf972.el9.noar 12/12 2024-09-17T23:03:04.931 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-17T23:03:05.537 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:01:08 ago on Tue 17 Sep 2024 11:01:57 PM UTC. 2024-09-17T23:03:05.677 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-09-17T23:03:05.678 INFO:teuthology.orchestra.run.smithi006.stdout:====================================================================================== 2024-09-17T23:03:05.678 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repository Size 2024-09-17T23:03:05.678 INFO:teuthology.orchestra.run.smithi006.stdout:====================================================================================== 2024-09-17T23:03:05.678 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-09-17T23:03:05.678 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-diskprediction-local noarch 2:19.1.1-330.gc2ddf972.el9 ceph-noarch 7.4 M 2024-09-17T23:03:05.678 INFO:teuthology.orchestra.run.smithi006.stdout:Installing dependencies: 2024-09-17T23:03:05.679 INFO:teuthology.orchestra.run.smithi006.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-17T23:03:05.679 INFO:teuthology.orchestra.run.smithi006.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-17T23:03:05.679 INFO:teuthology.orchestra.run.smithi006.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-17T23:03:05.679 INFO:teuthology.orchestra.run.smithi006.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-17T23:03:05.679 INFO:teuthology.orchestra.run.smithi006.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-17T23:03:05.679 INFO:teuthology.orchestra.run.smithi006.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-17T23:03:05.679 INFO:teuthology.orchestra.run.smithi006.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-17T23:03:05.679 INFO:teuthology.orchestra.run.smithi006.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-17T23:03:05.679 INFO:teuthology.orchestra.run.smithi006.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-17T23:03:05.679 INFO:teuthology.orchestra.run.smithi006.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-17T23:03:05.679 INFO:teuthology.orchestra.run.smithi006.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-17T23:03:05.679 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:03:05.679 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-09-17T23:03:05.679 INFO:teuthology.orchestra.run.smithi006.stdout:====================================================================================== 2024-09-17T23:03:05.679 INFO:teuthology.orchestra.run.smithi006.stdout:Install 12 Packages 2024-09-17T23:03:05.680 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:03:05.680 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 38 M 2024-09-17T23:03:05.680 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 212 M 2024-09-17T23:03:05.680 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-09-17T23:03:06.160 INFO:teuthology.orchestra.run.smithi006.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.0 MB/s | 191 kB 00:00 2024-09-17T23:03:06.180 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.1.1-330.gc2ddf972.el9.noar 12/12 2024-09-17T23:03:06.180 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-grafana-dashboards-2:19.1.1-330.gc2ddf972.el9 1/12 2024-09-17T23:03:06.180 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mgr-dashboard-2:19.1.1-330.gc2ddf972.el9.noar 2/12 2024-09-17T23:03:06.180 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-prometheus-alerts-2:19.1.1-330.gc2ddf972.el9. 3/12 2024-09-17T23:03:06.180 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-17T23:03:06.180 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-17T23:03:06.180 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-17T23:03:06.180 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-17T23:03:06.181 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-17T23:03:06.181 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-17T23:03:06.181 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-17T23:03:06.181 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-17T23:03:06.202 INFO:teuthology.orchestra.run.smithi006.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 3.5 MB/s | 801 kB 00:00 2024-09-17T23:03:06.286 INFO:teuthology.orchestra.run.smithi006.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 261 kB/s | 33 kB 00:00 2024-09-17T23:03:06.336 INFO:teuthology.orchestra.run.smithi006.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 366 kB/s | 18 kB 00:00 2024-09-17T23:03:06.436 INFO:teuthology.orchestra.run.smithi006.stdout:(5/12): ceph-mgr-diskprediction-local-19.1.1-33 16 MB/s | 7.4 MB 00:00 2024-09-17T23:03:06.461 INFO:teuthology.orchestra.run.smithi006.stdout:(6/12): openblas-0.3.26-2.el9.x86_64.rpm 307 kB/s | 38 kB 00:00 2024-09-17T23:03:06.506 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-17T23:03:06.506 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:03:06.506 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-09-17T23:03:06.506 INFO:teuthology.orchestra.run.smithi038.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-17T23:03:06.506 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-grafana-dashboards-2:19.1.1-330.gc2ddf972.el9.noarch 2024-09-17T23:03:06.506 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-dashboard-2:19.1.1-330.gc2ddf972.el9.noarch 2024-09-17T23:03:06.506 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-prometheus-alerts-2:19.1.1-330.gc2ddf972.el9.noarch 2024-09-17T23:03:06.506 INFO:teuthology.orchestra.run.smithi038.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-17T23:03:06.506 INFO:teuthology.orchestra.run.smithi038.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-17T23:03:06.506 INFO:teuthology.orchestra.run.smithi038.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-17T23:03:06.506 INFO:teuthology.orchestra.run.smithi038.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-17T23:03:06.506 INFO:teuthology.orchestra.run.smithi038.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-17T23:03:06.507 INFO:teuthology.orchestra.run.smithi038.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-17T23:03:06.507 INFO:teuthology.orchestra.run.smithi038.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-17T23:03:06.507 INFO:teuthology.orchestra.run.smithi038.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-17T23:03:06.507 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:03:06.507 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-09-17T23:03:06.570 INFO:teuthology.orchestra.run.smithi006.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 2.2 MB/s | 245 kB 00:00 2024-09-17T23:03:06.735 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-17T23:03:07.012 INFO:teuthology.orchestra.run.smithi006.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 3.7 MB/s | 3.0 MB 00:00 2024-09-17T23:03:07.095 INFO:teuthology.orchestra.run.smithi006.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.9 MB/s | 250 kB 00:00 2024-09-17T23:03:07.345 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:07 ago on Tue 17 Sep 2024 11:02:00 PM UTC. 2024-09-17T23:03:07.484 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-09-17T23:03:07.485 INFO:teuthology.orchestra.run.smithi038.stdout:====================================================================================== 2024-09-17T23:03:07.485 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repository Size 2024-09-17T23:03:07.486 INFO:teuthology.orchestra.run.smithi038.stdout:====================================================================================== 2024-09-17T23:03:07.486 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-09-17T23:03:07.486 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-diskprediction-local noarch 2:19.1.1-330.gc2ddf972.el9 ceph-noarch 7.4 M 2024-09-17T23:03:07.486 INFO:teuthology.orchestra.run.smithi038.stdout:Installing dependencies: 2024-09-17T23:03:07.486 INFO:teuthology.orchestra.run.smithi038.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-17T23:03:07.486 INFO:teuthology.orchestra.run.smithi038.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-17T23:03:07.486 INFO:teuthology.orchestra.run.smithi038.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-17T23:03:07.486 INFO:teuthology.orchestra.run.smithi038.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-17T23:03:07.486 INFO:teuthology.orchestra.run.smithi038.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-17T23:03:07.486 INFO:teuthology.orchestra.run.smithi038.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-17T23:03:07.486 INFO:teuthology.orchestra.run.smithi038.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-17T23:03:07.486 INFO:teuthology.orchestra.run.smithi038.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-17T23:03:07.486 INFO:teuthology.orchestra.run.smithi038.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-17T23:03:07.487 INFO:teuthology.orchestra.run.smithi038.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-17T23:03:07.487 INFO:teuthology.orchestra.run.smithi038.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-17T23:03:07.487 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:03:07.487 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-09-17T23:03:07.487 INFO:teuthology.orchestra.run.smithi038.stdout:====================================================================================== 2024-09-17T23:03:07.487 INFO:teuthology.orchestra.run.smithi038.stdout:Install 12 Packages 2024-09-17T23:03:07.487 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:03:07.487 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 38 M 2024-09-17T23:03:07.488 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 212 M 2024-09-17T23:03:07.488 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-09-17T23:03:07.604 INFO:teuthology.orchestra.run.smithi006.stdout:(10/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 4.2 MB/s | 4.9 MB 00:01 2024-09-17T23:03:07.696 INFO:teuthology.orchestra.run.smithi006.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 4.5 MB/s | 5.1 MB 00:01 2024-09-17T23:03:08.023 INFO:teuthology.orchestra.run.smithi038.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.6 MB/s | 191 kB 00:00 2024-09-17T23:03:08.056 INFO:teuthology.orchestra.run.smithi038.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 5.1 MB/s | 801 kB 00:00 2024-09-17T23:03:08.140 INFO:teuthology.orchestra.run.smithi038.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 279 kB/s | 33 kB 00:00 2024-09-17T23:03:08.190 INFO:teuthology.orchestra.run.smithi038.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 366 kB/s | 18 kB 00:00 2024-09-17T23:03:08.240 INFO:teuthology.orchestra.run.smithi038.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 771 kB/s | 38 kB 00:00 2024-09-17T23:03:08.366 INFO:teuthology.orchestra.run.smithi038.stdout:(6/12): ceph-mgr-diskprediction-local-19.1.1-33 16 MB/s | 7.4 MB 00:00 2024-09-17T23:03:08.533 INFO:teuthology.orchestra.run.smithi038.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 1.4 MB/s | 245 kB 00:00 2024-09-17T23:03:08.900 INFO:teuthology.orchestra.run.smithi038.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 3.5 MB/s | 3.0 MB 00:00 2024-09-17T23:03:08.992 INFO:teuthology.orchestra.run.smithi038.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.7 MB/s | 250 kB 00:00 2024-09-17T23:03:09.392 INFO:teuthology.orchestra.run.smithi038.stdout:(10/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 4.3 MB/s | 4.9 MB 00:01 2024-09-17T23:03:09.504 INFO:teuthology.orchestra.run.smithi006.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 6.7 MB/s | 16 MB 00:02 2024-09-17T23:03:09.505 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-09-17T23:03:09.506 INFO:teuthology.orchestra.run.smithi006.stdout:Total 10 MB/s | 38 MB 00:03 2024-09-17T23:03:09.625 INFO:teuthology.orchestra.run.smithi038.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 4.7 MB/s | 5.1 MB 00:01 2024-09-17T23:03:09.876 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-09-17T23:03:09.904 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-09-17T23:03:09.904 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-09-17T23:03:10.363 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-09-17T23:03:10.363 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-09-17T23:03:11.150 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-09-17T23:03:11.212 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-17T23:03:11.253 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-17T23:03:11.358 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-17T23:03:11.396 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-17T23:03:11.589 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-17T23:03:11.619 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-17T23:03:11.692 INFO:teuthology.orchestra.run.smithi038.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 6.0 MB/s | 16 MB 00:02 2024-09-17T23:03:11.693 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-09-17T23:03:11.694 INFO:teuthology.orchestra.run.smithi038.stdout:Total 9.1 MB/s | 38 MB 00:04 2024-09-17T23:03:11.698 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-17T23:03:12.067 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-09-17T23:03:12.095 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-09-17T23:03:12.095 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-09-17T23:03:12.398 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-17T23:03:12.521 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-17T23:03:12.550 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-09-17T23:03:12.551 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-09-17T23:03:13.350 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-09-17T23:03:13.433 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-17T23:03:13.491 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-17T23:03:13.600 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-17T23:03:13.642 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-17T23:03:13.842 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-17T23:03:13.854 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-17T23:03:13.882 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-17T23:03:13.961 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-17T23:03:14.692 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-17T23:03:14.817 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-17T23:03:14.933 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-17T23:03:14.968 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mgr-diskprediction-local-2:19.1.1-330.gc2ddf9 12/12 2024-09-17T23:03:16.162 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-17T23:03:17.244 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-17T23:03:17.273 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mgr-diskprediction-local-2:19.1.1-330.gc2ddf9 12/12 2024-09-17T23:03:17.645 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.1.1-330.gc2ddf9 12/12 2024-09-17T23:03:17.645 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.1.1-330.gc2ddf9 1/12 2024-09-17T23:03:17.645 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-17T23:03:17.645 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-17T23:03:17.645 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-17T23:03:17.645 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-17T23:03:17.645 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-17T23:03:17.646 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-17T23:03:17.646 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-17T23:03:17.646 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-17T23:03:17.646 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-17T23:03:17.646 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-17T23:03:18.042 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-17T23:03:18.042 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:03:18.042 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-09-17T23:03:18.042 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-diskprediction-local-2:19.1.1-330.gc2ddf972.el9.noarch 2024-09-17T23:03:18.042 INFO:teuthology.orchestra.run.smithi006.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-17T23:03:18.042 INFO:teuthology.orchestra.run.smithi006.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-17T23:03:18.042 INFO:teuthology.orchestra.run.smithi006.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-17T23:03:18.042 INFO:teuthology.orchestra.run.smithi006.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-17T23:03:18.042 INFO:teuthology.orchestra.run.smithi006.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-17T23:03:18.043 INFO:teuthology.orchestra.run.smithi006.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-17T23:03:18.043 INFO:teuthology.orchestra.run.smithi006.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-17T23:03:18.043 INFO:teuthology.orchestra.run.smithi006.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-17T23:03:18.043 INFO:teuthology.orchestra.run.smithi006.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-17T23:03:18.043 INFO:teuthology.orchestra.run.smithi006.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-17T23:03:18.043 INFO:teuthology.orchestra.run.smithi006.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-17T23:03:18.043 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:03:18.043 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-09-17T23:03:18.435 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install ceph-mgr-rook 2024-09-17T23:03:19.036 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:01:22 ago on Tue 17 Sep 2024 11:01:57 PM UTC. 2024-09-17T23:03:19.172 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-09-17T23:03:19.173 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-09-17T23:03:19.173 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repository Size 2024-09-17T23:03:19.174 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-09-17T23:03:19.174 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-09-17T23:03:19.174 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-rook noarch 2:19.1.1-330.gc2ddf972.el9 ceph-noarch 49 k 2024-09-17T23:03:19.174 INFO:teuthology.orchestra.run.smithi006.stdout:Installing dependencies: 2024-09-17T23:03:19.174 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-17T23:03:19.174 INFO:teuthology.orchestra.run.smithi006.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-17T23:03:19.174 INFO:teuthology.orchestra.run.smithi006.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-17T23:03:19.174 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-17T23:03:19.174 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-17T23:03:19.174 INFO:teuthology.orchestra.run.smithi006.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-17T23:03:19.174 INFO:teuthology.orchestra.run.smithi006.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-17T23:03:19.174 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-17T23:03:19.175 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-17T23:03:19.175 INFO:teuthology.orchestra.run.smithi006.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-17T23:03:19.175 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-17T23:03:19.175 INFO:teuthology.orchestra.run.smithi006.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-17T23:03:19.175 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:03:19.175 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-09-17T23:03:19.175 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-09-17T23:03:19.175 INFO:teuthology.orchestra.run.smithi006.stdout:Install 13 Packages 2024-09-17T23:03:19.175 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:03:19.176 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 2.2 M 2024-09-17T23:03:19.176 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 26 M 2024-09-17T23:03:19.176 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-09-17T23:03:19.728 INFO:teuthology.orchestra.run.smithi006.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 165 kB/s | 19 kB 00:00 2024-09-17T23:03:19.753 INFO:teuthology.orchestra.run.smithi006.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 181 kB/s | 26 kB 00:00 2024-09-17T23:03:19.778 INFO:teuthology.orchestra.run.smithi006.stdout:(3/13): ceph-mgr-rook-19.1.1-330.gc2ddf972.el9. 296 kB/s | 49 kB 00:00 2024-09-17T23:03:19.837 INFO:teuthology.orchestra.run.smithi006.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.0 MB/s | 222 kB 00:00 2024-09-17T23:03:19.862 INFO:teuthology.orchestra.run.smithi006.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.4 MB/s | 159 kB 00:00 2024-09-17T23:03:19.887 INFO:teuthology.orchestra.run.smithi006.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 54 kB 00:00 2024-09-17T23:03:19.931 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.1.1-330.gc2ddf9 12/12 2024-09-17T23:03:19.931 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.1.1-330.gc2ddf9 1/12 2024-09-17T23:03:19.931 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-17T23:03:19.931 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-17T23:03:19.931 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-17T23:03:19.931 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-17T23:03:19.931 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-17T23:03:19.931 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-17T23:03:19.931 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-17T23:03:19.931 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-17T23:03:19.932 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-17T23:03:19.932 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-17T23:03:19.938 INFO:teuthology.orchestra.run.smithi006.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 429 kB/s | 32 kB 00:00 2024-09-17T23:03:19.971 INFO:teuthology.orchestra.run.smithi006.stdout:(8/13): python3-certifi-2023.05.07-4.el9.noarch 188 kB/s | 14 kB 00:00 2024-09-17T23:03:19.996 INFO:teuthology.orchestra.run.smithi006.stdout:(9/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.2 MB/s | 279 kB 00:00 2024-09-17T23:03:20.038 INFO:teuthology.orchestra.run.smithi006.stdout:(10/13): python3-google-auth-2.34.0-1.el9.noarc 2.2 MB/s | 220 kB 00:00 2024-09-17T23:03:20.081 INFO:teuthology.orchestra.run.smithi006.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 707 kB/s | 59 kB 00:00 2024-09-17T23:03:20.106 INFO:teuthology.orchestra.run.smithi006.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.4 MB/s | 90 kB 00:00 2024-09-17T23:03:20.164 INFO:teuthology.orchestra.run.smithi006.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 5.3 MB/s | 1.0 MB 00:00 2024-09-17T23:03:20.168 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-09-17T23:03:20.168 INFO:teuthology.orchestra.run.smithi006.stdout:Total 2.2 MB/s | 2.2 MB 00:00 2024-09-17T23:03:20.237 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-09-17T23:03:20.253 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-09-17T23:03:20.253 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-09-17T23:03:20.333 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-17T23:03:20.333 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:03:20.333 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-09-17T23:03:20.333 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-diskprediction-local-2:19.1.1-330.gc2ddf972.el9.noarch 2024-09-17T23:03:20.333 INFO:teuthology.orchestra.run.smithi038.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-17T23:03:20.333 INFO:teuthology.orchestra.run.smithi038.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-17T23:03:20.334 INFO:teuthology.orchestra.run.smithi038.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-17T23:03:20.334 INFO:teuthology.orchestra.run.smithi038.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-17T23:03:20.334 INFO:teuthology.orchestra.run.smithi038.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-17T23:03:20.334 INFO:teuthology.orchestra.run.smithi038.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-17T23:03:20.334 INFO:teuthology.orchestra.run.smithi038.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-17T23:03:20.334 INFO:teuthology.orchestra.run.smithi038.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-17T23:03:20.334 INFO:teuthology.orchestra.run.smithi038.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-17T23:03:20.334 INFO:teuthology.orchestra.run.smithi038.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-17T23:03:20.334 INFO:teuthology.orchestra.run.smithi038.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-17T23:03:20.334 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:03:20.334 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-09-17T23:03:20.354 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-09-17T23:03:20.356 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-09-17T23:03:20.751 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-mgr-rook 2024-09-17T23:03:21.011 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-09-17T23:03:21.196 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-17T23:03:21.260 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-17T23:03:21.324 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-17T23:03:21.358 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:21 ago on Tue 17 Sep 2024 11:02:00 PM UTC. 2024-09-17T23:03:21.359 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-17T23:03:21.401 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-17T23:03:21.498 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-09-17T23:03:21.499 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-17T23:03:21.499 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repository Size 2024-09-17T23:03:21.499 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-17T23:03:21.499 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-09-17T23:03:21.499 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-rook noarch 2:19.1.1-330.gc2ddf972.el9 ceph-noarch 49 k 2024-09-17T23:03:21.500 INFO:teuthology.orchestra.run.smithi038.stdout:Installing dependencies: 2024-09-17T23:03:21.500 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-17T23:03:21.500 INFO:teuthology.orchestra.run.smithi038.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-17T23:03:21.500 INFO:teuthology.orchestra.run.smithi038.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-17T23:03:21.500 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-17T23:03:21.500 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-17T23:03:21.500 INFO:teuthology.orchestra.run.smithi038.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-17T23:03:21.500 INFO:teuthology.orchestra.run.smithi038.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-17T23:03:21.500 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-17T23:03:21.500 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-17T23:03:21.500 INFO:teuthology.orchestra.run.smithi038.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-17T23:03:21.500 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-17T23:03:21.501 INFO:teuthology.orchestra.run.smithi038.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-17T23:03:21.501 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:03:21.501 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-09-17T23:03:21.501 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-17T23:03:21.501 INFO:teuthology.orchestra.run.smithi038.stdout:Install 13 Packages 2024-09-17T23:03:21.501 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:03:21.501 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 2.2 M 2024-09-17T23:03:21.501 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 26 M 2024-09-17T23:03:21.502 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-09-17T23:03:21.517 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-17T23:03:21.644 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-17T23:03:21.695 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-17T23:03:21.766 INFO:teuthology.orchestra.run.smithi038.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 197 kB/s | 26 kB 00:00 2024-09-17T23:03:21.791 INFO:teuthology.orchestra.run.smithi038.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 123 kB/s | 19 kB 00:00 2024-09-17T23:03:21.817 INFO:teuthology.orchestra.run.smithi038.stdout:(3/13): ceph-mgr-rook-19.1.1-330.gc2ddf972.el9. 273 kB/s | 49 kB 00:00 2024-09-17T23:03:21.925 INFO:teuthology.orchestra.run.smithi038.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.4 MB/s | 222 kB 00:00 2024-09-17T23:03:21.950 INFO:teuthology.orchestra.run.smithi038.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.0 MB/s | 159 kB 00:00 2024-09-17T23:03:21.984 INFO:teuthology.orchestra.run.smithi038.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 917 kB/s | 54 kB 00:00 2024-09-17T23:03:22.001 INFO:teuthology.orchestra.run.smithi038.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 638 kB/s | 32 kB 00:00 2024-09-17T23:03:22.043 INFO:teuthology.orchestra.run.smithi038.stdout:(8/13): python3-certifi-2023.05.07-4.el9.noarch 241 kB/s | 14 kB 00:00 2024-09-17T23:03:22.068 INFO:teuthology.orchestra.run.smithi038.stdout:(9/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.1 MB/s | 279 kB 00:00 2024-09-17T23:03:22.093 INFO:teuthology.orchestra.run.smithi038.stdout:(10/13): python3-google-auth-2.34.0-1.el9.noarc 2.5 MB/s | 220 kB 00:00 2024-09-17T23:03:22.144 INFO:teuthology.orchestra.run.smithi038.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 2.0 MB/s | 90 kB 00:00 2024-09-17T23:03:22.169 INFO:teuthology.orchestra.run.smithi038.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 588 kB/s | 59 kB 00:00 2024-09-17T23:03:22.236 INFO:teuthology.orchestra.run.smithi038.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 5.3 MB/s | 1.0 MB 00:00 2024-09-17T23:03:22.239 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-09-17T23:03:22.239 INFO:teuthology.orchestra.run.smithi038.stdout:Total 3.0 MB/s | 2.2 MB 00:00 2024-09-17T23:03:22.305 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-09-17T23:03:22.322 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-09-17T23:03:22.322 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-09-17T23:03:22.423 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-09-17T23:03:22.424 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-09-17T23:03:22.459 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-17T23:03:22.516 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-17T23:03:22.549 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-17T23:03:22.599 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-17T23:03:22.627 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mgr-rook-2:19.1.1-330.gc2ddf972.el9.noarch 13/13 2024-09-17T23:03:23.074 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-09-17T23:03:23.253 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-17T23:03:23.323 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-17T23:03:23.387 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-17T23:03:23.422 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-17T23:03:23.461 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-mgr-rook-2:19.1.1-330.gc2ddf972.el9.noarch 13/13 2024-09-17T23:03:23.461 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mgr-rook-2:19.1.1-330.gc2ddf972.el9.noarch 1/13 2024-09-17T23:03:23.462 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-17T23:03:23.462 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-17T23:03:23.462 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-17T23:03:23.462 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-17T23:03:23.462 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-17T23:03:23.462 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-17T23:03:23.462 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-17T23:03:23.462 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-17T23:03:23.462 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-17T23:03:23.462 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-17T23:03:23.462 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-17T23:03:23.464 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-17T23:03:23.571 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-17T23:03:23.782 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-17T23:03:23.859 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-17T23:03:23.925 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-17T23:03:23.925 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:03:23.925 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-09-17T23:03:23.925 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-rook-2:19.1.1-330.gc2ddf972.el9.noarch 2024-09-17T23:03:23.925 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-17T23:03:23.925 INFO:teuthology.orchestra.run.smithi006.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-17T23:03:23.925 INFO:teuthology.orchestra.run.smithi006.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-17T23:03:23.925 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-17T23:03:23.925 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-17T23:03:23.925 INFO:teuthology.orchestra.run.smithi006.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-17T23:03:23.925 INFO:teuthology.orchestra.run.smithi006.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-17T23:03:23.925 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-17T23:03:23.925 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-17T23:03:23.925 INFO:teuthology.orchestra.run.smithi006.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-17T23:03:23.926 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-17T23:03:23.926 INFO:teuthology.orchestra.run.smithi006.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-17T23:03:23.926 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:03:23.926 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-09-17T23:03:24.309 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install ceph-mgr-cephadm 2024-09-17T23:03:24.635 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-17T23:03:24.688 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-17T23:03:24.730 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-17T23:03:24.779 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-17T23:03:24.807 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mgr-rook-2:19.1.1-330.gc2ddf972.el9.noarch 13/13 2024-09-17T23:03:24.917 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:01:27 ago on Tue 17 Sep 2024 11:01:57 PM UTC. 2024-09-17T23:03:25.055 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-09-17T23:03:25.056 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-09-17T23:03:25.056 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repository Size 2024-09-17T23:03:25.056 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-09-17T23:03:25.056 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-09-17T23:03:25.056 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-cephadm noarch 2:19.1.1-330.gc2ddf972.el9 ceph-noarch 146 k 2024-09-17T23:03:25.056 INFO:teuthology.orchestra.run.smithi006.stdout:Installing dependencies: 2024-09-17T23:03:25.056 INFO:teuthology.orchestra.run.smithi006.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-17T23:03:25.056 INFO:teuthology.orchestra.run.smithi006.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-17T23:03:25.056 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-17T23:03:25.056 INFO:teuthology.orchestra.run.smithi006.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-17T23:03:25.057 INFO:teuthology.orchestra.run.smithi006.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-17T23:03:25.057 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:03:25.057 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-09-17T23:03:25.057 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-09-17T23:03:25.057 INFO:teuthology.orchestra.run.smithi006.stdout:Install 6 Packages 2024-09-17T23:03:25.057 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:03:25.058 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 7.0 M 2024-09-17T23:03:25.058 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 33 M 2024-09-17T23:03:25.058 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-09-17T23:03:25.560 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mgr-rook-2:19.1.1-330.gc2ddf972.el9.noarch 13/13 2024-09-17T23:03:25.560 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mgr-rook-2:19.1.1-330.gc2ddf972.el9.noarch 1/13 2024-09-17T23:03:25.560 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-17T23:03:25.561 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-17T23:03:25.561 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-17T23:03:25.561 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-17T23:03:25.561 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-17T23:03:25.561 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-17T23:03:25.561 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-17T23:03:25.561 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-17T23:03:25.561 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-17T23:03:25.561 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-17T23:03:25.561 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-17T23:03:25.667 INFO:teuthology.orchestra.run.smithi006.stdout:(1/6): ceph-mgr-cephadm-19.1.1-330.gc2ddf972.el 688 kB/s | 146 kB 00:00 2024-09-17T23:03:25.801 INFO:teuthology.orchestra.run.smithi006.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 4.0 MB/s | 548 kB 00:00 2024-09-17T23:03:25.834 INFO:teuthology.orchestra.run.smithi006.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 1.7 MB/s | 58 kB 00:00 2024-09-17T23:03:25.868 INFO:teuthology.orchestra.run.smithi006.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 2.2 MB/s | 74 kB 00:00 2024-09-17T23:03:25.904 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-17T23:03:25.904 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:03:25.904 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-09-17T23:03:25.904 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-rook-2:19.1.1-330.gc2ddf972.el9.noarch 2024-09-17T23:03:25.904 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-17T23:03:25.904 INFO:teuthology.orchestra.run.smithi038.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-17T23:03:25.905 INFO:teuthology.orchestra.run.smithi038.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-17T23:03:25.905 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-17T23:03:25.905 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-17T23:03:25.905 INFO:teuthology.orchestra.run.smithi038.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-17T23:03:25.905 INFO:teuthology.orchestra.run.smithi038.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-17T23:03:25.905 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-17T23:03:25.905 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-17T23:03:25.905 INFO:teuthology.orchestra.run.smithi038.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-17T23:03:25.905 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-17T23:03:25.905 INFO:teuthology.orchestra.run.smithi038.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-17T23:03:25.905 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:03:25.905 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-09-17T23:03:26.102 INFO:teuthology.orchestra.run.smithi006.stdout:(5/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 384 kB/s | 248 kB 00:00 2024-09-17T23:03:26.160 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-mgr-cephadm 2024-09-17T23:03:26.768 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:26 ago on Tue 17 Sep 2024 11:02:00 PM UTC. 2024-09-17T23:03:26.906 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-09-17T23:03:26.907 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-17T23:03:26.907 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repository Size 2024-09-17T23:03:26.907 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-17T23:03:26.907 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-09-17T23:03:26.907 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-cephadm noarch 2:19.1.1-330.gc2ddf972.el9 ceph-noarch 146 k 2024-09-17T23:03:26.907 INFO:teuthology.orchestra.run.smithi038.stdout:Installing dependencies: 2024-09-17T23:03:26.907 INFO:teuthology.orchestra.run.smithi038.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-17T23:03:26.907 INFO:teuthology.orchestra.run.smithi038.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-17T23:03:26.907 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-17T23:03:26.907 INFO:teuthology.orchestra.run.smithi038.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-17T23:03:26.907 INFO:teuthology.orchestra.run.smithi038.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-17T23:03:26.907 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:03:26.907 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-09-17T23:03:26.908 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-17T23:03:26.908 INFO:teuthology.orchestra.run.smithi038.stdout:Install 6 Packages 2024-09-17T23:03:26.908 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:03:26.908 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 7.0 M 2024-09-17T23:03:26.908 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 33 M 2024-09-17T23:03:26.908 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-09-17T23:03:27.717 INFO:teuthology.orchestra.run.smithi038.stdout:(1/6): ceph-mgr-cephadm-19.1.1-330.gc2ddf972.el 728 kB/s | 146 kB 00:00 2024-09-17T23:03:27.739 INFO:teuthology.orchestra.run.smithi038.stdout:(2/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.1 MB/s | 248 kB 00:00 2024-09-17T23:03:27.797 INFO:teuthology.orchestra.run.smithi038.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 1.1 MB/s | 58 kB 00:00 2024-09-17T23:03:27.822 INFO:teuthology.orchestra.run.smithi038.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 2.9 MB/s | 74 kB 00:00 2024-09-17T23:03:27.856 INFO:teuthology.orchestra.run.smithi038.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 3.8 MB/s | 548 kB 00:00 2024-09-17T23:03:27.956 INFO:teuthology.orchestra.run.smithi038.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 13 MB/s | 6.0 MB 00:00 2024-09-17T23:03:27.959 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-09-17T23:03:27.960 INFO:teuthology.orchestra.run.smithi038.stdout:Total 6.7 MB/s | 7.0 MB 00:01 2024-09-17T23:03:28.035 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-09-17T23:03:28.048 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-09-17T23:03:28.048 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-09-17T23:03:28.154 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-09-17T23:03:28.154 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-09-17T23:03:28.483 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-09-17T23:03:28.661 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-17T23:03:28.709 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-17T23:03:28.868 INFO:teuthology.orchestra.run.smithi006.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 1.7 MB/s | 6.0 MB 00:03 2024-09-17T23:03:28.870 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-09-17T23:03:28.871 INFO:teuthology.orchestra.run.smithi006.stdout:Total 1.8 MB/s | 7.0 MB 00:03 2024-09-17T23:03:28.946 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-09-17T23:03:28.958 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-09-17T23:03:28.959 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-09-17T23:03:29.065 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-09-17T23:03:29.066 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-09-17T23:03:29.261 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-17T23:03:29.348 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-17T23:03:29.404 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-09-17T23:03:29.427 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-17T23:03:29.462 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mgr-cephadm-2:19.1.1-330.gc2ddf972.el9.noarch 6/6 2024-09-17T23:03:29.584 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-17T23:03:29.646 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-17T23:03:30.216 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-17T23:03:30.277 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.1.1-330.gc2ddf972.el9.noarch 6/6 2024-09-17T23:03:30.277 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mgr-cephadm-2:19.1.1-330.gc2ddf972.el9.noarch 1/6 2024-09-17T23:03:30.277 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-17T23:03:30.277 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-17T23:03:30.277 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-17T23:03:30.277 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-17T23:03:30.309 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-17T23:03:30.398 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-17T23:03:30.424 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mgr-cephadm-2:19.1.1-330.gc2ddf972.el9.noarch 6/6 2024-09-17T23:03:30.545 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-17T23:03:30.545 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:03:30.545 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-09-17T23:03:30.545 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-cephadm-2:19.1.1-330.gc2ddf972.el9.noarch 2024-09-17T23:03:30.545 INFO:teuthology.orchestra.run.smithi038.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-17T23:03:30.545 INFO:teuthology.orchestra.run.smithi038.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-17T23:03:30.545 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-17T23:03:30.546 INFO:teuthology.orchestra.run.smithi038.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-17T23:03:30.546 INFO:teuthology.orchestra.run.smithi038.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-17T23:03:30.546 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:03:30.546 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-09-17T23:03:30.797 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-fuse 2024-09-17T23:03:31.189 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.1.1-330.gc2ddf972.el9.noarch 6/6 2024-09-17T23:03:31.189 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mgr-cephadm-2:19.1.1-330.gc2ddf972.el9.noarch 1/6 2024-09-17T23:03:31.189 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-17T23:03:31.189 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-17T23:03:31.189 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-17T23:03:31.189 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-17T23:03:31.407 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:31 ago on Tue 17 Sep 2024 11:02:00 PM UTC. 2024-09-17T23:03:31.505 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-17T23:03:31.505 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:03:31.505 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-09-17T23:03:31.505 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-cephadm-2:19.1.1-330.gc2ddf972.el9.noarch 2024-09-17T23:03:31.505 INFO:teuthology.orchestra.run.smithi006.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-17T23:03:31.505 INFO:teuthology.orchestra.run.smithi006.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-17T23:03:31.505 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-17T23:03:31.505 INFO:teuthology.orchestra.run.smithi006.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-17T23:03:31.505 INFO:teuthology.orchestra.run.smithi006.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-17T23:03:31.506 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:03:31.506 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-09-17T23:03:31.547 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-09-17T23:03:31.547 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-17T23:03:31.547 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repository Size 2024-09-17T23:03:31.547 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-17T23:03:31.547 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-09-17T23:03:31.547 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-fuse x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 897 k 2024-09-17T23:03:31.547 INFO:teuthology.orchestra.run.smithi038.stdout:Installing dependencies: 2024-09-17T23:03:31.547 INFO:teuthology.orchestra.run.smithi038.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-17T23:03:31.547 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:03:31.547 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-09-17T23:03:31.547 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-17T23:03:31.547 INFO:teuthology.orchestra.run.smithi038.stdout:Install 2 Packages 2024-09-17T23:03:31.548 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:03:31.548 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 977 k 2024-09-17T23:03:31.548 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 2.9 M 2024-09-17T23:03:31.548 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-09-17T23:03:31.741 INFO:teuthology.orchestra.run.smithi038.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 541 kB/s | 80 kB 00:00 2024-09-17T23:03:31.775 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install ceph-fuse 2024-09-17T23:03:31.875 INFO:teuthology.orchestra.run.smithi038.stdout:(2/2): ceph-fuse-19.1.1-330.gc2ddf972.el9.x86_6 3.1 MB/s | 897 kB 00:00 2024-09-17T23:03:31.875 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-09-17T23:03:31.876 INFO:teuthology.orchestra.run.smithi038.stdout:Total 2.9 MB/s | 977 kB 00:00 2024-09-17T23:03:31.888 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-09-17T23:03:31.910 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-09-17T23:03:31.910 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-09-17T23:03:31.996 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-09-17T23:03:31.997 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-09-17T23:03:32.199 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-09-17T23:03:32.363 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-17T23:03:32.376 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:01:35 ago on Tue 17 Sep 2024 11:01:57 PM UTC. 2024-09-17T23:03:32.435 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-fuse-2:19.1.1-330.gc2ddf972.el9.x86_64 2/2 2024-09-17T23:03:32.515 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-09-17T23:03:32.516 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-09-17T23:03:32.516 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repository Size 2024-09-17T23:03:32.516 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-09-17T23:03:32.516 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-09-17T23:03:32.516 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-fuse x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 897 k 2024-09-17T23:03:32.516 INFO:teuthology.orchestra.run.smithi006.stdout:Installing dependencies: 2024-09-17T23:03:32.517 INFO:teuthology.orchestra.run.smithi006.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-17T23:03:32.517 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:03:32.517 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-09-17T23:03:32.517 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-09-17T23:03:32.517 INFO:teuthology.orchestra.run.smithi006.stdout:Install 2 Packages 2024-09-17T23:03:32.517 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:03:32.517 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 977 k 2024-09-17T23:03:32.517 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 2.9 M 2024-09-17T23:03:32.518 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-09-17T23:03:33.053 INFO:teuthology.orchestra.run.smithi006.stdout:(1/2): ceph-fuse-19.1.1-330.gc2ddf972.el9.x86_6 3.1 MB/s | 897 kB 00:00 2024-09-17T23:03:33.078 INFO:teuthology.orchestra.run.smithi006.stdout:(2/2): fuse-2.9.9-16.el9.x86_64.rpm 261 kB/s | 80 kB 00:00 2024-09-17T23:03:33.079 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-09-17T23:03:33.079 INFO:teuthology.orchestra.run.smithi006.stdout:Total 1.7 MB/s | 977 kB 00:00 2024-09-17T23:03:33.091 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-09-17T23:03:33.115 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-09-17T23:03:33.115 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-09-17T23:03:33.166 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-fuse-2:19.1.1-330.gc2ddf972.el9.x86_64 2/2 2024-09-17T23:03:33.166 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-fuse-2:19.1.1-330.gc2ddf972.el9.x86_64 1/2 2024-09-17T23:03:33.202 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-09-17T23:03:33.202 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-09-17T23:03:33.395 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-09-17T23:03:33.446 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-17T23:03:33.446 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:03:33.446 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-09-17T23:03:33.446 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-fuse-2:19.1.1-330.gc2ddf972.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-17T23:03:33.446 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:03:33.446 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-09-17T23:03:33.540 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-17T23:03:33.614 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-fuse-2:19.1.1-330.gc2ddf972.el9.x86_64 2/2 2024-09-17T23:03:33.704 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-volume 2024-09-17T23:03:34.278 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-fuse-2:19.1.1-330.gc2ddf972.el9.x86_64 2/2 2024-09-17T23:03:34.278 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-fuse-2:19.1.1-330.gc2ddf972.el9.x86_64 1/2 2024-09-17T23:03:34.320 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:34 ago on Tue 17 Sep 2024 11:02:00 PM UTC. 2024-09-17T23:03:34.461 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-09-17T23:03:34.461 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-17T23:03:34.461 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repository Size 2024-09-17T23:03:34.462 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-17T23:03:34.462 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-09-17T23:03:34.462 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-volume noarch 2:19.1.1-330.gc2ddf972.el9 ceph-noarch 285 k 2024-09-17T23:03:34.462 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:03:34.462 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-09-17T23:03:34.462 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-17T23:03:34.462 INFO:teuthology.orchestra.run.smithi038.stdout:Install 1 Package 2024-09-17T23:03:34.462 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:03:34.462 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 285 k 2024-09-17T23:03:34.462 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 1.3 M 2024-09-17T23:03:34.462 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-09-17T23:03:34.558 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-17T23:03:34.558 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:03:34.558 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-09-17T23:03:34.558 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-fuse-2:19.1.1-330.gc2ddf972.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-17T23:03:34.558 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:03:34.559 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-09-17T23:03:34.688 INFO:teuthology.orchestra.run.smithi038.stdout:ceph-volume-19.1.1-330.gc2ddf972.el9.noarch.rpm 1.2 MB/s | 285 kB 00:00 2024-09-17T23:03:34.689 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-09-17T23:03:34.689 INFO:teuthology.orchestra.run.smithi038.stdout:Total 1.2 MB/s | 285 kB 00:00 2024-09-17T23:03:34.689 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-09-17T23:03:34.699 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-09-17T23:03:34.699 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-09-17T23:03:34.725 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-09-17T23:03:34.725 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-09-17T23:03:34.850 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install ceph-volume 2024-09-17T23:03:34.955 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-09-17T23:03:35.032 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-volume-2:19.1.1-330.gc2ddf972.el9.noarch 1/1 2024-09-17T23:03:35.049 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-volume-2:19.1.1-330.gc2ddf972.el9.noarch 1/1 2024-09-17T23:03:35.049 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T23:03:35.049 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-17T23:03:35.049 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:03:35.463 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:01:38 ago on Tue 17 Sep 2024 11:01:57 PM UTC. 2024-09-17T23:03:35.603 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-09-17T23:03:35.604 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-09-17T23:03:35.604 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repository Size 2024-09-17T23:03:35.604 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-09-17T23:03:35.604 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-09-17T23:03:35.604 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-volume noarch 2:19.1.1-330.gc2ddf972.el9 ceph-noarch 285 k 2024-09-17T23:03:35.604 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:03:35.604 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-09-17T23:03:35.604 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-09-17T23:03:35.604 INFO:teuthology.orchestra.run.smithi006.stdout:Install 1 Package 2024-09-17T23:03:35.604 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:03:35.604 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 285 k 2024-09-17T23:03:35.604 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 1.3 M 2024-09-17T23:03:35.605 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-09-17T23:03:35.817 INFO:teuthology.orchestra.run.smithi006.stdout:ceph-volume-19.1.1-330.gc2ddf972.el9.noarch.rpm 1.3 MB/s | 285 kB 00:00 2024-09-17T23:03:35.817 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-09-17T23:03:35.818 INFO:teuthology.orchestra.run.smithi006.stdout:Total 1.3 MB/s | 285 kB 00:00 2024-09-17T23:03:35.818 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-09-17T23:03:35.827 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-09-17T23:03:35.828 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-09-17T23:03:35.861 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-09-17T23:03:35.861 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-09-17T23:03:36.093 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-volume-2:19.1.1-330.gc2ddf972.el9.noarch 1/1 2024-09-17T23:03:36.093 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:03:36.093 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-09-17T23:03:36.093 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-volume-2:19.1.1-330.gc2ddf972.el9.noarch 2024-09-17T23:03:36.093 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:03:36.094 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-09-17T23:03:36.095 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-09-17T23:03:36.161 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-volume-2:19.1.1-330.gc2ddf972.el9.noarch 1/1 2024-09-17T23:03:36.178 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-volume-2:19.1.1-330.gc2ddf972.el9.noarch 1/1 2024-09-17T23:03:36.178 INFO:teuthology.orchestra.run.smithi006.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T23:03:36.178 INFO:teuthology.orchestra.run.smithi006.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-17T23:03:36.178 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:03:36.392 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install librados-devel 2024-09-17T23:03:37.007 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:37 ago on Tue 17 Sep 2024 11:02:00 PM UTC. 2024-09-17T23:03:37.146 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-09-17T23:03:37.147 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-17T23:03:37.147 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repo Size 2024-09-17T23:03:37.147 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-17T23:03:37.147 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-09-17T23:03:37.147 INFO:teuthology.orchestra.run.smithi038.stdout: librados-devel x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 127 k 2024-09-17T23:03:37.147 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:03:37.147 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-09-17T23:03:37.147 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-17T23:03:37.147 INFO:teuthology.orchestra.run.smithi038.stdout:Install 1 Package 2024-09-17T23:03:37.147 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:03:37.147 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 127 k 2024-09-17T23:03:37.148 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 456 k 2024-09-17T23:03:37.148 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-09-17T23:03:37.204 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-volume-2:19.1.1-330.gc2ddf972.el9.noarch 1/1 2024-09-17T23:03:37.204 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:03:37.204 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-09-17T23:03:37.204 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-volume-2:19.1.1-330.gc2ddf972.el9.noarch 2024-09-17T23:03:37.204 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:03:37.205 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-09-17T23:03:37.360 INFO:teuthology.orchestra.run.smithi038.stdout:librados-devel-19.1.1-330.gc2ddf972.el9.x86_64. 595 kB/s | 127 kB 00:00 2024-09-17T23:03:37.361 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-09-17T23:03:37.361 INFO:teuthology.orchestra.run.smithi038.stdout:Total 592 kB/s | 127 kB 00:00 2024-09-17T23:03:37.361 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-09-17T23:03:37.367 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-09-17T23:03:37.367 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-09-17T23:03:37.434 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-09-17T23:03:37.434 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-09-17T23:03:37.528 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install librados-devel 2024-09-17T23:03:37.606 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-09-17T23:03:37.723 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : librados-devel-2:19.1.1-330.gc2ddf972.el9.x86_64 1/1 2024-09-17T23:03:38.143 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:01:41 ago on Tue 17 Sep 2024 11:01:57 PM UTC. 2024-09-17T23:03:38.224 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: librados-devel-2:19.1.1-330.gc2ddf972.el9.x86_64 1/1 2024-09-17T23:03:38.282 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-09-17T23:03:38.282 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-09-17T23:03:38.282 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repo Size 2024-09-17T23:03:38.282 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-09-17T23:03:38.283 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-09-17T23:03:38.283 INFO:teuthology.orchestra.run.smithi006.stdout: librados-devel x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 127 k 2024-09-17T23:03:38.283 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:03:38.283 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-09-17T23:03:38.283 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-09-17T23:03:38.283 INFO:teuthology.orchestra.run.smithi006.stdout:Install 1 Package 2024-09-17T23:03:38.283 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:03:38.283 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 127 k 2024-09-17T23:03:38.283 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 456 k 2024-09-17T23:03:38.283 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-09-17T23:03:38.495 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librados-devel-2:19.1.1-330.gc2ddf972.el9.x86_64 1/1 2024-09-17T23:03:38.495 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:03:38.495 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-09-17T23:03:38.495 INFO:teuthology.orchestra.run.smithi038.stdout: librados-devel-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:03:38.495 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:03:38.495 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-09-17T23:03:38.497 INFO:teuthology.orchestra.run.smithi006.stdout:librados-devel-19.1.1-330.gc2ddf972.el9.x86_64. 592 kB/s | 127 kB 00:00 2024-09-17T23:03:38.497 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-09-17T23:03:38.498 INFO:teuthology.orchestra.run.smithi006.stdout:Total 588 kB/s | 127 kB 00:00 2024-09-17T23:03:38.498 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-09-17T23:03:38.504 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-09-17T23:03:38.504 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-09-17T23:03:38.572 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-09-17T23:03:38.572 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-09-17T23:03:38.716 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install libcephfs2 2024-09-17T23:03:38.727 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-09-17T23:03:38.832 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : librados-devel-2:19.1.1-330.gc2ddf972.el9.x86_64 1/1 2024-09-17T23:03:39.256 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: librados-devel-2:19.1.1-330.gc2ddf972.el9.x86_64 1/1 2024-09-17T23:03:39.332 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:39 ago on Tue 17 Sep 2024 11:02:00 PM UTC. 2024-09-17T23:03:39.425 INFO:teuthology.orchestra.run.smithi038.stdout:Package libcephfs2-2:19.1.1-330.gc2ddf972.el9.x86_64 is already installed. 2024-09-17T23:03:39.472 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-09-17T23:03:39.473 INFO:teuthology.orchestra.run.smithi038.stdout:Nothing to do. 2024-09-17T23:03:39.473 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-09-17T23:03:39.531 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librados-devel-2:19.1.1-330.gc2ddf972.el9.x86_64 1/1 2024-09-17T23:03:39.531 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:03:39.531 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-09-17T23:03:39.531 INFO:teuthology.orchestra.run.smithi006.stdout: librados-devel-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:03:39.531 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:03:39.531 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-09-17T23:03:39.535 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install libcephfs-devel 2024-09-17T23:03:39.901 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install libcephfs2 2024-09-17T23:03:40.141 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:40 ago on Tue 17 Sep 2024 11:02:00 PM UTC. 2024-09-17T23:03:40.279 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-09-17T23:03:40.280 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-17T23:03:40.280 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repo Size 2024-09-17T23:03:40.280 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-17T23:03:40.280 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-09-17T23:03:40.280 INFO:teuthology.orchestra.run.smithi038.stdout: libcephfs-devel x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 32 k 2024-09-17T23:03:40.280 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:03:40.280 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-09-17T23:03:40.280 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-17T23:03:40.280 INFO:teuthology.orchestra.run.smithi038.stdout:Install 1 Package 2024-09-17T23:03:40.281 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:03:40.281 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 32 k 2024-09-17T23:03:40.281 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 142 k 2024-09-17T23:03:40.281 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-09-17T23:03:40.439 INFO:teuthology.orchestra.run.smithi038.stdout:libcephfs-devel-19.1.1-330.gc2ddf972.el9.x86_64 200 kB/s | 32 kB 00:00 2024-09-17T23:03:40.440 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-09-17T23:03:40.440 INFO:teuthology.orchestra.run.smithi038.stdout:Total 198 kB/s | 32 kB 00:00 2024-09-17T23:03:40.440 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-09-17T23:03:40.444 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-09-17T23:03:40.444 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-09-17T23:03:40.462 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-09-17T23:03:40.462 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-09-17T23:03:40.514 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:01:43 ago on Tue 17 Sep 2024 11:01:57 PM UTC. 2024-09-17T23:03:40.558 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-09-17T23:03:40.608 INFO:teuthology.orchestra.run.smithi006.stdout:Package libcephfs2-2:19.1.1-330.gc2ddf972.el9.x86_64 is already installed. 2024-09-17T23:03:40.654 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-09-17T23:03:40.655 INFO:teuthology.orchestra.run.smithi006.stdout:Nothing to do. 2024-09-17T23:03:40.655 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-09-17T23:03:40.665 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libcephfs-devel-2:19.1.1-330.gc2ddf972.el9.x86_64 1/1 2024-09-17T23:03:40.716 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install libcephfs-devel 2024-09-17T23:03:41.046 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: libcephfs-devel-2:19.1.1-330.gc2ddf972.el9.x86_64 1/1 2024-09-17T23:03:41.274 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libcephfs-devel-2:19.1.1-330.gc2ddf972.el9.x86_64 1/1 2024-09-17T23:03:41.274 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:03:41.275 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-09-17T23:03:41.275 INFO:teuthology.orchestra.run.smithi038.stdout: libcephfs-devel-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:03:41.275 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:03:41.275 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-09-17T23:03:41.323 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:01:44 ago on Tue 17 Sep 2024 11:01:57 PM UTC. 2024-09-17T23:03:41.460 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-09-17T23:03:41.460 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-09-17T23:03:41.460 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repo Size 2024-09-17T23:03:41.461 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-09-17T23:03:41.461 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-09-17T23:03:41.461 INFO:teuthology.orchestra.run.smithi006.stdout: libcephfs-devel x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 32 k 2024-09-17T23:03:41.461 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:03:41.461 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-09-17T23:03:41.461 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-09-17T23:03:41.461 INFO:teuthology.orchestra.run.smithi006.stdout:Install 1 Package 2024-09-17T23:03:41.461 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:03:41.461 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 32 k 2024-09-17T23:03:41.461 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 142 k 2024-09-17T23:03:41.461 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-09-17T23:03:41.463 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install librados2 2024-09-17T23:03:41.734 INFO:teuthology.orchestra.run.smithi006.stdout:libcephfs-devel-19.1.1-330.gc2ddf972.el9.x86_64 116 kB/s | 32 kB 00:00 2024-09-17T23:03:41.735 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-09-17T23:03:41.735 INFO:teuthology.orchestra.run.smithi006.stdout:Total 116 kB/s | 32 kB 00:00 2024-09-17T23:03:41.735 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-09-17T23:03:41.738 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-09-17T23:03:41.739 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-09-17T23:03:41.759 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-09-17T23:03:41.760 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-09-17T23:03:41.853 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-09-17T23:03:42.065 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:42 ago on Tue 17 Sep 2024 11:02:00 PM UTC. 2024-09-17T23:03:42.092 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libcephfs-devel-2:19.1.1-330.gc2ddf972.el9.x86_64 1/1 2024-09-17T23:03:42.158 INFO:teuthology.orchestra.run.smithi038.stdout:Package librados2-2:19.1.1-330.gc2ddf972.el9.x86_64 is already installed. 2024-09-17T23:03:42.205 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-09-17T23:03:42.206 INFO:teuthology.orchestra.run.smithi038.stdout:Nothing to do. 2024-09-17T23:03:42.206 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-09-17T23:03:42.273 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install librbd1 2024-09-17T23:03:42.472 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: libcephfs-devel-2:19.1.1-330.gc2ddf972.el9.x86_64 1/1 2024-09-17T23:03:42.719 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libcephfs-devel-2:19.1.1-330.gc2ddf972.el9.x86_64 1/1 2024-09-17T23:03:42.719 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:03:42.719 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-09-17T23:03:42.720 INFO:teuthology.orchestra.run.smithi006.stdout: libcephfs-devel-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:03:42.720 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:03:42.720 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-09-17T23:03:42.875 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:42 ago on Tue 17 Sep 2024 11:02:00 PM UTC. 2024-09-17T23:03:42.914 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install librados2 2024-09-17T23:03:42.967 INFO:teuthology.orchestra.run.smithi038.stdout:Package librbd1-2:19.1.1-330.gc2ddf972.el9.x86_64 is already installed. 2024-09-17T23:03:43.013 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-09-17T23:03:43.014 INFO:teuthology.orchestra.run.smithi038.stdout:Nothing to do. 2024-09-17T23:03:43.014 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-09-17T23:03:43.084 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install python3-rados 2024-09-17T23:03:43.514 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:01:46 ago on Tue 17 Sep 2024 11:01:57 PM UTC. 2024-09-17T23:03:43.606 INFO:teuthology.orchestra.run.smithi006.stdout:Package librados2-2:19.1.1-330.gc2ddf972.el9.x86_64 is already installed. 2024-09-17T23:03:43.653 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-09-17T23:03:43.654 INFO:teuthology.orchestra.run.smithi006.stdout:Nothing to do. 2024-09-17T23:03:43.654 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-09-17T23:03:43.693 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:43 ago on Tue 17 Sep 2024 11:02:00 PM UTC. 2024-09-17T23:03:43.714 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install librbd1 2024-09-17T23:03:43.791 INFO:teuthology.orchestra.run.smithi038.stdout:Package python3-rados-2:19.1.1-330.gc2ddf972.el9.x86_64 is already installed. 2024-09-17T23:03:43.836 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-09-17T23:03:43.837 INFO:teuthology.orchestra.run.smithi038.stdout:Nothing to do. 2024-09-17T23:03:43.837 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-09-17T23:03:43.900 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install python3-rgw 2024-09-17T23:03:44.317 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:01:47 ago on Tue 17 Sep 2024 11:01:57 PM UTC. 2024-09-17T23:03:44.411 INFO:teuthology.orchestra.run.smithi006.stdout:Package librbd1-2:19.1.1-330.gc2ddf972.el9.x86_64 is already installed. 2024-09-17T23:03:44.457 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-09-17T23:03:44.458 INFO:teuthology.orchestra.run.smithi006.stdout:Nothing to do. 2024-09-17T23:03:44.458 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-09-17T23:03:44.514 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:44 ago on Tue 17 Sep 2024 11:02:00 PM UTC. 2024-09-17T23:03:44.529 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install python3-rados 2024-09-17T23:03:44.607 INFO:teuthology.orchestra.run.smithi038.stdout:Package python3-rgw-2:19.1.1-330.gc2ddf972.el9.x86_64 is already installed. 2024-09-17T23:03:44.653 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-09-17T23:03:44.654 INFO:teuthology.orchestra.run.smithi038.stdout:Nothing to do. 2024-09-17T23:03:44.654 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-09-17T23:03:44.727 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install python3-cephfs 2024-09-17T23:03:45.136 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:01:48 ago on Tue 17 Sep 2024 11:01:57 PM UTC. 2024-09-17T23:03:45.229 INFO:teuthology.orchestra.run.smithi006.stdout:Package python3-rados-2:19.1.1-330.gc2ddf972.el9.x86_64 is already installed. 2024-09-17T23:03:45.275 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-09-17T23:03:45.276 INFO:teuthology.orchestra.run.smithi006.stdout:Nothing to do. 2024-09-17T23:03:45.276 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-09-17T23:03:45.333 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:45 ago on Tue 17 Sep 2024 11:02:00 PM UTC. 2024-09-17T23:03:45.345 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install python3-rgw 2024-09-17T23:03:45.424 INFO:teuthology.orchestra.run.smithi038.stdout:Package python3-cephfs-2:19.1.1-330.gc2ddf972.el9.x86_64 is already installed. 2024-09-17T23:03:45.469 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-09-17T23:03:45.470 INFO:teuthology.orchestra.run.smithi038.stdout:Nothing to do. 2024-09-17T23:03:45.470 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-09-17T23:03:45.533 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install python3-rbd 2024-09-17T23:03:45.974 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:01:48 ago on Tue 17 Sep 2024 11:01:57 PM UTC. 2024-09-17T23:03:46.072 INFO:teuthology.orchestra.run.smithi006.stdout:Package python3-rgw-2:19.1.1-330.gc2ddf972.el9.x86_64 is already installed. 2024-09-17T23:03:46.118 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-09-17T23:03:46.119 INFO:teuthology.orchestra.run.smithi006.stdout:Nothing to do. 2024-09-17T23:03:46.119 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-09-17T23:03:46.145 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:46 ago on Tue 17 Sep 2024 11:02:00 PM UTC. 2024-09-17T23:03:46.192 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install python3-cephfs 2024-09-17T23:03:46.237 INFO:teuthology.orchestra.run.smithi038.stdout:Package python3-rbd-2:19.1.1-330.gc2ddf972.el9.x86_64 is already installed. 2024-09-17T23:03:46.282 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-09-17T23:03:46.283 INFO:teuthology.orchestra.run.smithi038.stdout:Nothing to do. 2024-09-17T23:03:46.283 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-09-17T23:03:46.343 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install rbd-fuse 2024-09-17T23:03:46.805 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:01:49 ago on Tue 17 Sep 2024 11:01:57 PM UTC. 2024-09-17T23:03:46.898 INFO:teuthology.orchestra.run.smithi006.stdout:Package python3-cephfs-2:19.1.1-330.gc2ddf972.el9.x86_64 is already installed. 2024-09-17T23:03:46.945 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-09-17T23:03:46.946 INFO:teuthology.orchestra.run.smithi006.stdout:Nothing to do. 2024-09-17T23:03:46.946 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-09-17T23:03:46.949 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:46 ago on Tue 17 Sep 2024 11:02:00 PM UTC. 2024-09-17T23:03:47.013 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install python3-rbd 2024-09-17T23:03:47.088 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-09-17T23:03:47.088 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-17T23:03:47.089 INFO:teuthology.orchestra.run.smithi038.stdout: Package Architecture Version Repository Size 2024-09-17T23:03:47.089 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-17T23:03:47.089 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-09-17T23:03:47.089 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-fuse x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 85 k 2024-09-17T23:03:47.089 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:03:47.089 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-09-17T23:03:47.089 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-17T23:03:47.089 INFO:teuthology.orchestra.run.smithi038.stdout:Install 1 Package 2024-09-17T23:03:47.089 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:03:47.089 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 85 k 2024-09-17T23:03:47.089 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 231 k 2024-09-17T23:03:47.089 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-09-17T23:03:47.443 INFO:teuthology.orchestra.run.smithi038.stdout:rbd-fuse-19.1.1-330.gc2ddf972.el9.x86_64.rpm 242 kB/s | 85 kB 00:00 2024-09-17T23:03:47.443 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-09-17T23:03:47.443 INFO:teuthology.orchestra.run.smithi038.stdout:Total 241 kB/s | 85 kB 00:00 2024-09-17T23:03:47.443 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-09-17T23:03:47.451 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-09-17T23:03:47.451 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-09-17T23:03:47.519 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-09-17T23:03:47.520 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-09-17T23:03:47.627 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:01:50 ago on Tue 17 Sep 2024 11:01:57 PM UTC. 2024-09-17T23:03:47.693 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-09-17T23:03:47.720 INFO:teuthology.orchestra.run.smithi006.stdout:Package python3-rbd-2:19.1.1-330.gc2ddf972.el9.x86_64 is already installed. 2024-09-17T23:03:47.766 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-09-17T23:03:47.768 INFO:teuthology.orchestra.run.smithi006.stdout:Nothing to do. 2024-09-17T23:03:47.768 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-09-17T23:03:47.809 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : rbd-fuse-2:19.1.1-330.gc2ddf972.el9.x86_64 1/1 2024-09-17T23:03:47.830 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install rbd-fuse 2024-09-17T23:03:48.199 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: rbd-fuse-2:19.1.1-330.gc2ddf972.el9.x86_64 1/1 2024-09-17T23:03:48.433 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:01:51 ago on Tue 17 Sep 2024 11:01:57 PM UTC. 2024-09-17T23:03:48.451 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : rbd-fuse-2:19.1.1-330.gc2ddf972.el9.x86_64 1/1 2024-09-17T23:03:48.452 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:03:48.452 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-09-17T23:03:48.452 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-fuse-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:03:48.452 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:03:48.452 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-09-17T23:03:48.572 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-09-17T23:03:48.573 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-09-17T23:03:48.573 INFO:teuthology.orchestra.run.smithi006.stdout: Package Architecture Version Repository Size 2024-09-17T23:03:48.573 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-09-17T23:03:48.573 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-09-17T23:03:48.573 INFO:teuthology.orchestra.run.smithi006.stdout: rbd-fuse x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 85 k 2024-09-17T23:03:48.573 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:03:48.573 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-09-17T23:03:48.573 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-09-17T23:03:48.573 INFO:teuthology.orchestra.run.smithi006.stdout:Install 1 Package 2024-09-17T23:03:48.573 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:03:48.573 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 85 k 2024-09-17T23:03:48.573 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 231 k 2024-09-17T23:03:48.574 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-09-17T23:03:48.715 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install rbd-mirror 2024-09-17T23:03:48.913 INFO:teuthology.orchestra.run.smithi006.stdout:rbd-fuse-19.1.1-330.gc2ddf972.el9.x86_64.rpm 252 kB/s | 85 kB 00:00 2024-09-17T23:03:48.914 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-09-17T23:03:48.914 INFO:teuthology.orchestra.run.smithi006.stdout:Total 251 kB/s | 85 kB 00:00 2024-09-17T23:03:48.914 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-09-17T23:03:48.922 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-09-17T23:03:48.922 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-09-17T23:03:48.990 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-09-17T23:03:48.990 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-09-17T23:03:49.156 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-09-17T23:03:49.255 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : rbd-fuse-2:19.1.1-330.gc2ddf972.el9.x86_64 1/1 2024-09-17T23:03:49.338 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:49 ago on Tue 17 Sep 2024 11:02:00 PM UTC. 2024-09-17T23:03:49.480 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-09-17T23:03:49.481 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-17T23:03:49.481 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repo Size 2024-09-17T23:03:49.481 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-17T23:03:49.481 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-09-17T23:03:49.481 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-mirror x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 3.1 M 2024-09-17T23:03:49.481 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:03:49.482 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-09-17T23:03:49.482 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-17T23:03:49.482 INFO:teuthology.orchestra.run.smithi038.stdout:Install 1 Package 2024-09-17T23:03:49.482 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:03:49.482 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 3.1 M 2024-09-17T23:03:49.482 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 13 M 2024-09-17T23:03:49.482 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-09-17T23:03:49.685 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: rbd-fuse-2:19.1.1-330.gc2ddf972.el9.x86_64 1/1 2024-09-17T23:03:49.944 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : rbd-fuse-2:19.1.1-330.gc2ddf972.el9.x86_64 1/1 2024-09-17T23:03:49.944 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:03:49.944 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-09-17T23:03:49.944 INFO:teuthology.orchestra.run.smithi006.stdout: rbd-fuse-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:03:49.944 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:03:49.944 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-09-17T23:03:50.164 INFO:teuthology.orchestra.run.smithi038.stdout:rbd-mirror-19.1.1-330.gc2ddf972.el9.x86_64.rpm 4.5 MB/s | 3.1 MB 00:00 2024-09-17T23:03:50.164 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-09-17T23:03:50.165 INFO:teuthology.orchestra.run.smithi038.stdout:Total 4.5 MB/s | 3.1 MB 00:00 2024-09-17T23:03:50.165 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-09-17T23:03:50.176 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-09-17T23:03:50.176 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-09-17T23:03:50.259 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install rbd-mirror 2024-09-17T23:03:50.262 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-09-17T23:03:50.262 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-09-17T23:03:50.716 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-09-17T23:03:50.781 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : rbd-mirror-2:19.1.1-330.gc2ddf972.el9.x86_64 1/1 2024-09-17T23:03:50.809 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: rbd-mirror-2:19.1.1-330.gc2ddf972.el9.x86_64 1/1 2024-09-17T23:03:50.809 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T23:03:50.809 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-17T23:03:50.809 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-17T23:03:50.809 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-17T23:03:50.809 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:03:50.876 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:01:53 ago on Tue 17 Sep 2024 11:01:57 PM UTC. 2024-09-17T23:03:51.016 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-09-17T23:03:51.017 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-09-17T23:03:51.017 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repo Size 2024-09-17T23:03:51.017 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-09-17T23:03:51.017 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-09-17T23:03:51.018 INFO:teuthology.orchestra.run.smithi006.stdout: rbd-mirror x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 3.1 M 2024-09-17T23:03:51.018 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:03:51.018 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-09-17T23:03:51.018 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-09-17T23:03:51.018 INFO:teuthology.orchestra.run.smithi006.stdout:Install 1 Package 2024-09-17T23:03:51.018 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:03:51.018 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 3.1 M 2024-09-17T23:03:51.018 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 13 M 2024-09-17T23:03:51.018 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-09-17T23:03:51.750 INFO:teuthology.orchestra.run.smithi006.stdout:rbd-mirror-19.1.1-330.gc2ddf972.el9.x86_64.rpm 4.2 MB/s | 3.1 MB 00:00 2024-09-17T23:03:51.750 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-09-17T23:03:51.751 INFO:teuthology.orchestra.run.smithi006.stdout:Total 4.2 MB/s | 3.1 MB 00:00 2024-09-17T23:03:51.751 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-09-17T23:03:51.761 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-09-17T23:03:51.761 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-09-17T23:03:51.849 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-09-17T23:03:51.849 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-09-17T23:03:51.982 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : rbd-mirror-2:19.1.1-330.gc2ddf972.el9.x86_64 1/1 2024-09-17T23:03:51.982 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:03:51.982 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-09-17T23:03:51.982 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-mirror-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:03:51.982 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:03:51.982 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-09-17T23:03:52.244 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install rbd-nbd 2024-09-17T23:03:52.304 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-09-17T23:03:52.367 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : rbd-mirror-2:19.1.1-330.gc2ddf972.el9.x86_64 1/1 2024-09-17T23:03:52.394 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: rbd-mirror-2:19.1.1-330.gc2ddf972.el9.x86_64 1/1 2024-09-17T23:03:52.394 INFO:teuthology.orchestra.run.smithi006.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T23:03:52.394 INFO:teuthology.orchestra.run.smithi006.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-17T23:03:52.394 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-17T23:03:52.394 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-17T23:03:52.394 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:03:52.856 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:52 ago on Tue 17 Sep 2024 11:02:00 PM UTC. 2024-09-17T23:03:52.994 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-09-17T23:03:52.995 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-17T23:03:52.995 INFO:teuthology.orchestra.run.smithi038.stdout: Package Architecture Version Repository Size 2024-09-17T23:03:52.995 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-17T23:03:52.995 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-09-17T23:03:52.995 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-nbd x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 170 k 2024-09-17T23:03:52.995 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:03:52.995 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-09-17T23:03:52.995 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-17T23:03:52.995 INFO:teuthology.orchestra.run.smithi038.stdout:Install 1 Package 2024-09-17T23:03:52.995 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:03:52.996 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 170 k 2024-09-17T23:03:52.996 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 494 k 2024-09-17T23:03:52.996 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-09-17T23:03:53.209 INFO:teuthology.orchestra.run.smithi038.stdout:rbd-nbd-19.1.1-330.gc2ddf972.el9.x86_64.rpm 801 kB/s | 170 kB 00:00 2024-09-17T23:03:53.210 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-09-17T23:03:53.210 INFO:teuthology.orchestra.run.smithi038.stdout:Total 795 kB/s | 170 kB 00:00 2024-09-17T23:03:53.210 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-09-17T23:03:53.218 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-09-17T23:03:53.218 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-09-17T23:03:53.287 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-09-17T23:03:53.287 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-09-17T23:03:53.449 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-09-17T23:03:53.545 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : rbd-mirror-2:19.1.1-330.gc2ddf972.el9.x86_64 1/1 2024-09-17T23:03:53.545 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:03:53.546 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-09-17T23:03:53.546 INFO:teuthology.orchestra.run.smithi006.stdout: rbd-mirror-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:03:53.546 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:03:53.546 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-09-17T23:03:53.560 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : rbd-nbd-2:19.1.1-330.gc2ddf972.el9.x86_64 1/1 2024-09-17T23:03:53.832 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install rbd-nbd 2024-09-17T23:03:54.000 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: rbd-nbd-2:19.1.1-330.gc2ddf972.el9.x86_64 1/1 2024-09-17T23:03:54.250 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : rbd-nbd-2:19.1.1-330.gc2ddf972.el9.x86_64 1/1 2024-09-17T23:03:54.250 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:03:54.251 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-09-17T23:03:54.251 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-nbd-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:03:54.251 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:03:54.251 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-09-17T23:03:54.441 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:01:57 ago on Tue 17 Sep 2024 11:01:57 PM UTC. 2024-09-17T23:03:54.479 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install rbd-mirror 2024-09-17T23:03:54.580 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-09-17T23:03:54.581 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-09-17T23:03:54.581 INFO:teuthology.orchestra.run.smithi006.stdout: Package Architecture Version Repository Size 2024-09-17T23:03:54.581 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-09-17T23:03:54.581 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-09-17T23:03:54.581 INFO:teuthology.orchestra.run.smithi006.stdout: rbd-nbd x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 170 k 2024-09-17T23:03:54.581 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:03:54.581 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-09-17T23:03:54.581 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-09-17T23:03:54.581 INFO:teuthology.orchestra.run.smithi006.stdout:Install 1 Package 2024-09-17T23:03:54.581 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:03:54.582 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 170 k 2024-09-17T23:03:54.582 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 494 k 2024-09-17T23:03:54.582 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-09-17T23:03:54.795 INFO:teuthology.orchestra.run.smithi006.stdout:rbd-nbd-19.1.1-330.gc2ddf972.el9.x86_64.rpm 799 kB/s | 170 kB 00:00 2024-09-17T23:03:54.796 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-09-17T23:03:54.796 INFO:teuthology.orchestra.run.smithi006.stdout:Total 793 kB/s | 170 kB 00:00 2024-09-17T23:03:54.796 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-09-17T23:03:54.805 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-09-17T23:03:54.805 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-09-17T23:03:54.877 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-09-17T23:03:54.877 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-09-17T23:03:55.037 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-09-17T23:03:55.094 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:55 ago on Tue 17 Sep 2024 11:02:00 PM UTC. 2024-09-17T23:03:55.139 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : rbd-nbd-2:19.1.1-330.gc2ddf972.el9.x86_64 1/1 2024-09-17T23:03:55.189 INFO:teuthology.orchestra.run.smithi038.stdout:Package rbd-mirror-2:19.1.1-330.gc2ddf972.el9.x86_64 is already installed. 2024-09-17T23:03:55.235 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-09-17T23:03:55.236 INFO:teuthology.orchestra.run.smithi038.stdout:Nothing to do. 2024-09-17T23:03:55.236 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-09-17T23:03:55.320 DEBUG:teuthology.parallel:result is None 2024-09-17T23:03:55.575 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: rbd-nbd-2:19.1.1-330.gc2ddf972.el9.x86_64 1/1 2024-09-17T23:03:55.870 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : rbd-nbd-2:19.1.1-330.gc2ddf972.el9.x86_64 1/1 2024-09-17T23:03:55.870 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:03:55.870 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-09-17T23:03:55.870 INFO:teuthology.orchestra.run.smithi006.stdout: rbd-nbd-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:03:55.870 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:03:55.871 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-09-17T23:03:56.106 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install rbd-mirror 2024-09-17T23:03:56.732 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:01:59 ago on Tue 17 Sep 2024 11:01:57 PM UTC. 2024-09-17T23:03:56.825 INFO:teuthology.orchestra.run.smithi006.stdout:Package rbd-mirror-2:19.1.1-330.gc2ddf972.el9.x86_64 is already installed. 2024-09-17T23:03:56.871 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-09-17T23:03:56.872 INFO:teuthology.orchestra.run.smithi006.stdout:Nothing to do. 2024-09-17T23:03:56.872 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-09-17T23:03:56.950 DEBUG:teuthology.parallel:result is None 2024-09-17T23:03:56.950 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=c2ddf9721ef254645201cdbabd9c8f49c30a586f 2024-09-17T23:03:57.100 DEBUG:teuthology.orchestra.run.smithi006:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-17T23:03:57.125 INFO:teuthology.orchestra.run.smithi006.stdout:19.1.1-330.gc2ddf972.el9 2024-09-17T23:03:57.126 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-330.gc2ddf972.el9 2024-09-17T23:03:57.126 INFO:teuthology.task.install:The correct ceph version 19.1.1-330.gc2ddf972 is installed. 2024-09-17T23:03:57.128 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=c2ddf9721ef254645201cdbabd9c8f49c30a586f 2024-09-17T23:03:57.266 DEBUG:teuthology.orchestra.run.smithi038:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-17T23:03:57.292 INFO:teuthology.orchestra.run.smithi038.stdout:19.1.1-330.gc2ddf972.el9 2024-09-17T23:03:57.292 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-330.gc2ddf972.el9 2024-09-17T23:03:57.292 INFO:teuthology.task.install:The correct ceph version 19.1.1-330.gc2ddf972 is installed. 2024-09-17T23:03:57.294 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-17T23:03:57.294 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-09-17T23:03:57.294 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-17T23:03:57.321 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-09-17T23:03:57.321 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-17T23:03:57.362 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-17T23:03:57.362 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-09-17T23:03:57.362 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/usr/bin/daemon-helper 2024-09-17T23:03:57.388 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-17T23:03:57.453 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-09-17T23:03:57.454 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/usr/bin/daemon-helper 2024-09-17T23:03:57.482 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-17T23:03:57.551 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-17T23:03:57.551 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-09-17T23:03:57.552 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-17T23:03:57.577 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-17T23:03:57.642 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-09-17T23:03:57.642 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-17T23:03:57.672 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-17T23:03:57.741 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-17T23:03:57.741 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-09-17T23:03:57.741 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/usr/bin/stdin-killer 2024-09-17T23:03:57.767 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-17T23:03:57.832 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-09-17T23:03:57.832 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/usr/bin/stdin-killer 2024-09-17T23:03:57.861 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-17T23:03:57.929 INFO:teuthology.run_tasks:Running task ceph... 2024-09-17T23:03:58.015 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-17T23:03:58.015 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 777 /var/log/ceph 2024-09-17T23:03:58.017 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod 777 /var/log/ceph 2024-09-17T23:03:58.044 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-17T23:03:58.045 DEBUG:teuthology.orchestra.run.smithi006:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-17T23:03:58.083 DEBUG:teuthology.orchestra.run.smithi038:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-17T23:03:58.112 INFO:tasks.ceph:Creating extra log directories... 2024-09-17T23:03:58.112 DEBUG:teuthology.orchestra.run.smithi006:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-17T23:03:58.148 DEBUG:teuthology.orchestra.run.smithi038:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-17T23:03:58.190 INFO:tasks.ceph:Creating ceph cluster cluster1... 2024-09-17T23:03:58.191 INFO:tasks.ceph:config {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'cluster1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-17T23:03:58.191 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-09-17_21:16:02-rbd-squid-distro-default-smithi/7909572', 'branch': 'squid', 'description': 'rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-zstd policy/none rbd-mirror/four-per-cluster supported-random-distro$/{centos_latest} workloads/rbd-mirror-snapshot-stress-workunit-fast-diff}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7909572', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'meta': [{'desc': 'run two ceph clusters and install rbd-mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': '2 ceph clusters with 1 mon and 3 osds each'}, {'desc': 'run four rbd-mirror daemons per cluster'}, {'desc': 'run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon'}], 'name': 'teuthology-2024-09-17_21:16:02-rbd-squid-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'squid'}, 'ceph': {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'c2ddf9721ef254645201cdbabd9c8f49c30a586f'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'c2ddf9721ef254645201cdbabd9c8f49c30a586f'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'squid', 'sha1': 'b0d356b98e08f6bc59ef15652f0d0e4298fc83c7'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 100, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'], ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6']], 'seed': 1983, 'sha1': 'c2ddf9721ef254645201cdbabd9c8f49c30a586f', 'sleep_before_teardown': 0, 'subset': '79/128', 'suite': 'rbd', 'suite_branch': 'squid', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_b0d356b98e08f6bc59ef15652f0d0e4298fc83c7/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': 'b0d356b98e08f6bc59ef15652f0d0e4298fc83c7', 'targets': {'smithi006.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBI25UHMjorqH5xXKrb+xjNLrKzHw3AquyMACLWNnQVzXSAKmfZF29gIWsRpW0CsJC8C7QZbDaT9NmJu+Vqwsk7o=', 'smithi038.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE1eTtc2bdOQzb/4YSSSQQ8mKGiFq5T8t3u2PuUJ7oED7RK6GtrnDZsTnuisjLHXzLmeaYHBzMxFHTTfoD/Zn6Y='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': 'c2ddf9721ef254645201cdbabd9c8f49c30a586f'}}, {'ceph': {'cluster': 'cluster1', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'c2ddf9721ef254645201cdbabd9c8f49c30a586f'}}, {'ceph': {'cluster': 'cluster2'}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.3', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.3', 'thrash': True}}, {'rbd-mirror-thrash': {'cluster': 'cluster1'}}, {'rbd-mirror-thrash': {'cluster': 'cluster2'}}, {'workunit': {'clients': {'cluster1.client.mirror': ['rbd/rbd_mirror_stress.sh']}, 'env': {'CEPH_ARGS': '', 'MIRROR_POOL_MODE': 'image', 'RBD_IMAGE_FEATURES': 'layering,exclusive-lock,object-map,fast-diff', 'RBD_MIRROR_INSTANCES': '4', 'RBD_MIRROR_MODE': 'snapshot', 'RBD_MIRROR_USE_EXISTING_CLUSTER': '1', 'RBD_MIRROR_USE_RBD_MIRROR': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-17_21:16:02', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-17T23:03:58.191 DEBUG:teuthology.orchestra.run.smithi006:> install -d -m0755 -- /home/ubuntu/cephtest/cluster1.data 2024-09-17T23:03:58.214 DEBUG:teuthology.orchestra.run.smithi038:> install -d -m0755 -- /home/ubuntu/cephtest/cluster1.data 2024-09-17T23:03:58.236 DEBUG:teuthology.orchestra.run.smithi006:> sudo install -d -m0777 -- /var/run/ceph 2024-09-17T23:03:58.271 DEBUG:teuthology.orchestra.run.smithi038:> sudo install -d -m0777 -- /var/run/ceph 2024-09-17T23:03:58.302 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-09-17T23:03:58.302 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/scratch_devs of=/dev/stdout 2024-09-17T23:03:58.354 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-17T23:03:58.354 DEBUG:teuthology.orchestra.run.smithi006:> stat /dev/vg_nvme/lv_1 2024-09-17T23:03:58.411 INFO:teuthology.orchestra.run.smithi006.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-17T23:03:58.411 INFO:teuthology.orchestra.run.smithi006.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T23:03:58.411 INFO:teuthology.orchestra.run.smithi006.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-09-17T23:03:58.411 INFO:teuthology.orchestra.run.smithi006.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T23:03:58.411 INFO:teuthology.orchestra.run.smithi006.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T23:03:58.411 INFO:teuthology.orchestra.run.smithi006.stdout:Access: 2024-09-17 23:03:52.651077272 +0000 2024-09-17T23:03:58.411 INFO:teuthology.orchestra.run.smithi006.stdout:Modify: 2024-09-17 23:01:10.880668405 +0000 2024-09-17T23:03:58.411 INFO:teuthology.orchestra.run.smithi006.stdout:Change: 2024-09-17 23:01:10.880668405 +0000 2024-09-17T23:03:58.411 INFO:teuthology.orchestra.run.smithi006.stdout: Birth: 2024-09-17 23:01:10.880668405 +0000 2024-09-17T23:03:58.412 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-17T23:03:58.475 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records in 2024-09-17T23:03:58.475 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records out 2024-09-17T23:03:58.476 INFO:teuthology.orchestra.run.smithi006.stderr:512 bytes copied, 0.000202629 s, 2.5 MB/s 2024-09-17T23:03:58.477 DEBUG:teuthology.orchestra.run.smithi006:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-17T23:03:58.534 DEBUG:teuthology.orchestra.run.smithi006:> stat /dev/vg_nvme/lv_2 2024-09-17T23:03:58.590 INFO:teuthology.orchestra.run.smithi006.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-17T23:03:58.590 INFO:teuthology.orchestra.run.smithi006.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T23:03:58.590 INFO:teuthology.orchestra.run.smithi006.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-09-17T23:03:58.590 INFO:teuthology.orchestra.run.smithi006.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T23:03:58.590 INFO:teuthology.orchestra.run.smithi006.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T23:03:58.590 INFO:teuthology.orchestra.run.smithi006.stdout:Access: 2024-09-17 23:03:52.652077262 +0000 2024-09-17T23:03:58.590 INFO:teuthology.orchestra.run.smithi006.stdout:Modify: 2024-09-17 23:01:11.218665071 +0000 2024-09-17T23:03:58.590 INFO:teuthology.orchestra.run.smithi006.stdout:Change: 2024-09-17 23:01:11.218665071 +0000 2024-09-17T23:03:58.590 INFO:teuthology.orchestra.run.smithi006.stdout: Birth: 2024-09-17 23:01:11.218665071 +0000 2024-09-17T23:03:58.591 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-17T23:03:58.653 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records in 2024-09-17T23:03:58.654 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records out 2024-09-17T23:03:58.654 INFO:teuthology.orchestra.run.smithi006.stderr:512 bytes copied, 0.00018919 s, 2.7 MB/s 2024-09-17T23:03:58.655 DEBUG:teuthology.orchestra.run.smithi006:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-17T23:03:58.711 DEBUG:teuthology.orchestra.run.smithi006:> stat /dev/vg_nvme/lv_3 2024-09-17T23:03:58.767 INFO:teuthology.orchestra.run.smithi006.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-17T23:03:58.767 INFO:teuthology.orchestra.run.smithi006.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T23:03:58.767 INFO:teuthology.orchestra.run.smithi006.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-09-17T23:03:58.767 INFO:teuthology.orchestra.run.smithi006.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T23:03:58.767 INFO:teuthology.orchestra.run.smithi006.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T23:03:58.767 INFO:teuthology.orchestra.run.smithi006.stdout:Access: 2024-09-17 23:03:52.652077262 +0000 2024-09-17T23:03:58.767 INFO:teuthology.orchestra.run.smithi006.stdout:Modify: 2024-09-17 23:01:11.554661756 +0000 2024-09-17T23:03:58.767 INFO:teuthology.orchestra.run.smithi006.stdout:Change: 2024-09-17 23:01:11.554661756 +0000 2024-09-17T23:03:58.767 INFO:teuthology.orchestra.run.smithi006.stdout: Birth: 2024-09-17 23:01:11.554661756 +0000 2024-09-17T23:03:58.767 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-17T23:03:58.831 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records in 2024-09-17T23:03:58.831 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records out 2024-09-17T23:03:58.831 INFO:teuthology.orchestra.run.smithi006.stderr:512 bytes copied, 0.000173083 s, 3.0 MB/s 2024-09-17T23:03:58.832 DEBUG:teuthology.orchestra.run.smithi006:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-17T23:03:58.888 DEBUG:teuthology.orchestra.run.smithi006:> stat /dev/vg_nvme/lv_4 2024-09-17T23:03:58.944 INFO:teuthology.orchestra.run.smithi006.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-17T23:03:58.944 INFO:teuthology.orchestra.run.smithi006.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T23:03:58.944 INFO:teuthology.orchestra.run.smithi006.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-09-17T23:03:58.944 INFO:teuthology.orchestra.run.smithi006.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T23:03:58.944 INFO:teuthology.orchestra.run.smithi006.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T23:03:58.944 INFO:teuthology.orchestra.run.smithi006.stdout:Access: 2024-09-17 23:03:52.653077252 +0000 2024-09-17T23:03:58.944 INFO:teuthology.orchestra.run.smithi006.stdout:Modify: 2024-09-17 23:01:11.881658530 +0000 2024-09-17T23:03:58.945 INFO:teuthology.orchestra.run.smithi006.stdout:Change: 2024-09-17 23:01:11.881658530 +0000 2024-09-17T23:03:58.945 INFO:teuthology.orchestra.run.smithi006.stdout: Birth: 2024-09-17 23:01:11.881658530 +0000 2024-09-17T23:03:58.945 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-17T23:03:59.008 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records in 2024-09-17T23:03:59.008 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records out 2024-09-17T23:03:59.008 INFO:teuthology.orchestra.run.smithi006.stderr:512 bytes copied, 0.000168743 s, 3.0 MB/s 2024-09-17T23:03:59.009 DEBUG:teuthology.orchestra.run.smithi006:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-17T23:03:59.065 INFO:tasks.ceph:osd dev map: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-17T23:03:59.066 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi006.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-17T23:03:59.066 INFO:tasks.ceph:Generating config... 2024-09-17T23:03:59.067 INFO:tasks.ceph:[client] debug rbd = 20 2024-09-17T23:03:59.067 INFO:tasks.ceph:[client] debug rbd_mirror = 15 2024-09-17T23:03:59.067 INFO:tasks.ceph:[client] log to stderr = False 2024-09-17T23:03:59.067 INFO:tasks.ceph:[client] rbd default features = 125 2024-09-17T23:03:59.068 INFO:tasks.ceph:[client] rbd mirror image policy type = none 2024-09-17T23:03:59.068 INFO:tasks.ceph:[client.mirror.0] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-17T23:03:59.068 INFO:tasks.ceph:[client.mirror.0] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-17T23:03:59.068 INFO:tasks.ceph:[client.mirror.1] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-17T23:03:59.068 INFO:tasks.ceph:[client.mirror.1] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-17T23:03:59.068 INFO:tasks.ceph:[client.mirror.2] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-17T23:03:59.068 INFO:tasks.ceph:[client.mirror.2] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-17T23:03:59.068 INFO:tasks.ceph:[client.mirror.3] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-17T23:03:59.068 INFO:tasks.ceph:[client.mirror.3] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-17T23:03:59.068 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-09-17T23:03:59.068 INFO:tasks.ceph:[global] mon warn on pool no app = False 2024-09-17T23:03:59.069 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-09-17T23:03:59.069 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-17T23:03:59.069 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-17T23:03:59.069 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-17T23:03:59.069 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-17T23:03:59.069 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-17T23:03:59.069 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-17T23:03:59.069 INFO:tasks.ceph:[osd] bluestore compression algorithm = zstd 2024-09-17T23:03:59.069 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-09-17T23:03:59.069 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-17T23:03:59.070 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-09-17T23:03:59.070 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-09-17T23:03:59.070 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-17T23:03:59.070 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-17T23:03:59.070 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-09-17T23:03:59.070 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-17T23:03:59.070 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-17T23:03:59.070 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-17T23:03:59.070 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-17T23:03:59.070 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-17T23:03:59.071 INFO:tasks.ceph:Setting up cluster1.mon.a... 2024-09-17T23:03:59.071 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/cluster1.keyring 2024-09-17T23:03:59.158 INFO:teuthology.orchestra.run.smithi006.stdout:creating /etc/ceph/cluster1.keyring 2024-09-17T23:03:59.160 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/cluster1.keyring 2024-09-17T23:03:59.204 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2024-09-17T23:03:59.270 DEBUG:tasks.ceph:Ceph mon addresses: [('cluster1.mon.a', '172.21.15.6')] 2024-09-17T23:03:59.270 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.6', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'mon.a': {}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}} 2024-09-17T23:03:59.271 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-09-17T23:03:59.271 DEBUG:teuthology.orchestra.run.smithi006:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-17T23:03:59.326 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.6 --print /home/ubuntu/cephtest/cluster1.monmap 2024-09-17T23:03:59.407 INFO:teuthology.orchestra.run.smithi006.stdout:monmaptool: monmap file /home/ubuntu/cephtest/cluster1.monmap 2024-09-17T23:03:59.407 INFO:teuthology.orchestra.run.smithi006.stdout:monmaptool: generated fsid 545dc37d-b29c-4f04-aad4-46458bcf9f65 2024-09-17T23:03:59.407 INFO:teuthology.orchestra.run.smithi006.stdout:setting min_mon_release = quincy 2024-09-17T23:03:59.407 INFO:teuthology.orchestra.run.smithi006.stdout:epoch 0 2024-09-17T23:03:59.407 INFO:teuthology.orchestra.run.smithi006.stdout:fsid 545dc37d-b29c-4f04-aad4-46458bcf9f65 2024-09-17T23:03:59.407 INFO:teuthology.orchestra.run.smithi006.stdout:last_changed 2024-09-17T23:03:59.405354+0000 2024-09-17T23:03:59.407 INFO:teuthology.orchestra.run.smithi006.stdout:created 2024-09-17T23:03:59.405354+0000 2024-09-17T23:03:59.407 INFO:teuthology.orchestra.run.smithi006.stdout:min_mon_release 17 (quincy) 2024-09-17T23:03:59.407 INFO:teuthology.orchestra.run.smithi006.stdout:election_strategy: 1 2024-09-17T23:03:59.407 INFO:teuthology.orchestra.run.smithi006.stdout:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2024-09-17T23:03:59.408 INFO:teuthology.orchestra.run.smithi006.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/cluster1.monmap (1 monitors) 2024-09-17T23:03:59.409 DEBUG:teuthology.orchestra.run.smithi006:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-17T23:03:59.463 INFO:tasks.ceph:Writing /etc/ceph/cluster1.conf for FSID 545dc37d-b29c-4f04-aad4-46458bcf9f65... 2024-09-17T23:03:59.464 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster1.conf && sudo chmod 0644 /etc/ceph/cluster1.conf > /dev/null 2024-09-17T23:03:59.506 DEBUG:teuthology.orchestra.run.smithi038:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster1.conf && sudo chmod 0644 /etc/ceph/cluster1.conf > /dev/null 2024-09-17T23:03:59.548 INFO:teuthology.orchestra.run.smithi006.stdout:[global] 2024-09-17T23:03:59.548 INFO:teuthology.orchestra.run.smithi006.stdout: chdir = "" 2024-09-17T23:03:59.548 INFO:teuthology.orchestra.run.smithi006.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-17T23:03:59.549 INFO:teuthology.orchestra.run.smithi006.stdout: auth supported = cephx 2024-09-17T23:03:59.549 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:03:59.549 INFO:teuthology.orchestra.run.smithi006.stdout: filestore xattr use omap = true 2024-09-17T23:03:59.549 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:03:59.549 INFO:teuthology.orchestra.run.smithi006.stdout: mon clock drift allowed = 1.000 2024-09-17T23:03:59.549 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:03:59.549 INFO:teuthology.orchestra.run.smithi006.stdout: osd crush chooseleaf type = 0 2024-09-17T23:03:59.549 INFO:teuthology.orchestra.run.smithi006.stdout: auth debug = true 2024-09-17T23:03:59.549 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:03:59.549 INFO:teuthology.orchestra.run.smithi006.stdout: ms die on old message = true 2024-09-17T23:03:59.549 INFO:teuthology.orchestra.run.smithi006.stdout: ms die on bug = true 2024-09-17T23:03:59.549 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:03:59.549 INFO:teuthology.orchestra.run.smithi006.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-17T23:03:59.550 INFO:teuthology.orchestra.run.smithi006.stdout: mon pg warn max object skew = 0 2024-09-17T23:03:59.550 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:03:59.550 INFO:teuthology.orchestra.run.smithi006.stdout: # disable pg_autoscaler by default for new pools 2024-09-17T23:03:59.550 INFO:teuthology.orchestra.run.smithi006.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-17T23:03:59.550 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:03:59.550 INFO:teuthology.orchestra.run.smithi006.stdout: osd pool default size = 2 2024-09-17T23:03:59.550 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:03:59.550 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd allow primary affinity = true 2024-09-17T23:03:59.550 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd allow pg remap = true 2024-09-17T23:03:59.550 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on legacy crush tunables = false 2024-09-17T23:03:59.550 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on crush straw calc version zero = false 2024-09-17T23:03:59.550 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on no sortbitwise = false 2024-09-17T23:03:59.550 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on osd down out interval zero = false 2024-09-17T23:03:59.550 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on too few osds = false 2024-09-17T23:03:59.551 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-17T23:03:59.551 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-17T23:03:59.551 INFO:teuthology.orchestra.run.smithi006.stdout: mon_allow_pool_size_one = true 2024-09-17T23:03:59.551 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:03:59.551 INFO:teuthology.orchestra.run.smithi006.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-17T23:03:59.551 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:03:59.551 INFO:teuthology.orchestra.run.smithi006.stdout: osd default data pool replay window = 5 2024-09-17T23:03:59.551 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:03:59.551 INFO:teuthology.orchestra.run.smithi006.stdout: mon allow pool delete = true 2024-09-17T23:03:59.551 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:03:59.551 INFO:teuthology.orchestra.run.smithi006.stdout: mon cluster log file level = debug 2024-09-17T23:03:59.551 INFO:teuthology.orchestra.run.smithi006.stdout: debug asserts on shutdown = true 2024-09-17T23:03:59.551 INFO:teuthology.orchestra.run.smithi006.stdout: mon health detail to clog = false 2024-09-17T23:03:59.551 INFO:teuthology.orchestra.run.smithi006.stdout: mon host = 172.21.15.6 2024-09-17T23:03:59.551 INFO:teuthology.orchestra.run.smithi006.stdout: mon client directed command retry = 5 2024-09-17T23:03:59.552 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on pool no app = False 2024-09-17T23:03:59.552 INFO:teuthology.orchestra.run.smithi006.stdout: ms inject socket failures = 5000 2024-09-17T23:03:59.552 INFO:teuthology.orchestra.run.smithi006.stdout: fsid = 545dc37d-b29c-4f04-aad4-46458bcf9f65 2024-09-17T23:03:59.552 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:03:59.552 INFO:teuthology.orchestra.run.smithi006.stdout:[osd] 2024-09-17T23:03:59.552 INFO:teuthology.orchestra.run.smithi006.stdout: osd journal size = 100 2024-09-17T23:03:59.552 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:03:59.552 INFO:teuthology.orchestra.run.smithi006.stdout: osd scrub load threshold = 5.0 2024-09-17T23:03:59.552 INFO:teuthology.orchestra.run.smithi006.stdout: osd scrub max interval = 600 2024-09-17T23:03:59.552 INFO:teuthology.orchestra.run.smithi006.stdout: osd mclock profile = high_recovery_ops 2024-09-17T23:03:59.552 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:03:59.552 INFO:teuthology.orchestra.run.smithi006.stdout: osd recover clone overlap = true 2024-09-17T23:03:59.552 INFO:teuthology.orchestra.run.smithi006.stdout: osd recovery max chunk = 1048576 2024-09-17T23:03:59.552 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:03:59.553 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug shutdown = true 2024-09-17T23:03:59.553 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug op order = true 2024-09-17T23:03:59.553 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug verify stray on activate = true 2024-09-17T23:03:59.553 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug trim objects = true 2024-09-17T23:03:59.553 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:03:59.553 INFO:teuthology.orchestra.run.smithi006.stdout: osd open classes on start = true 2024-09-17T23:03:59.553 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug pg log writeout = true 2024-09-17T23:03:59.553 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:03:59.553 INFO:teuthology.orchestra.run.smithi006.stdout: osd deep scrub update digest min age = 30 2024-09-17T23:03:59.553 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:03:59.553 INFO:teuthology.orchestra.run.smithi006.stdout: osd map max advance = 10 2024-09-17T23:03:59.553 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:03:59.553 INFO:teuthology.orchestra.run.smithi006.stdout: journal zero on create = true 2024-09-17T23:03:59.553 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:03:59.554 INFO:teuthology.orchestra.run.smithi006.stdout: filestore ondisk finisher threads = 3 2024-09-17T23:03:59.554 INFO:teuthology.orchestra.run.smithi006.stdout: filestore apply finisher threads = 3 2024-09-17T23:03:59.554 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:03:59.554 INFO:teuthology.orchestra.run.smithi006.stdout: bdev debug aio = true 2024-09-17T23:03:59.554 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug misdirected ops = true 2024-09-17T23:03:59.554 INFO:teuthology.orchestra.run.smithi006.stdout: bluestore block size = 96636764160 2024-09-17T23:03:59.554 INFO:teuthology.orchestra.run.smithi006.stdout: bluestore compression algorithm = zstd 2024-09-17T23:03:59.554 INFO:teuthology.orchestra.run.smithi006.stdout: bluestore compression mode = aggressive 2024-09-17T23:03:59.554 INFO:teuthology.orchestra.run.smithi006.stdout: bluestore fsck on mount = True 2024-09-17T23:03:59.554 INFO:teuthology.orchestra.run.smithi006.stdout: debug bluefs = 1/20 2024-09-17T23:03:59.554 INFO:teuthology.orchestra.run.smithi006.stdout: debug bluestore = 1/20 2024-09-17T23:03:59.554 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-09-17T23:03:59.554 INFO:teuthology.orchestra.run.smithi006.stdout: debug osd = 20 2024-09-17T23:03:59.554 INFO:teuthology.orchestra.run.smithi006.stdout: debug rocksdb = 4/10 2024-09-17T23:03:59.555 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd backfillfull_ratio = 0.85 2024-09-17T23:03:59.555 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd full ratio = 0.9 2024-09-17T23:03:59.555 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd nearfull ratio = 0.8 2024-09-17T23:03:59.555 INFO:teuthology.orchestra.run.smithi006.stdout: osd failsafe full ratio = 0.95 2024-09-17T23:03:59.555 INFO:teuthology.orchestra.run.smithi006.stdout: osd objectstore = bluestore 2024-09-17T23:03:59.555 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:03:59.555 INFO:teuthology.orchestra.run.smithi006.stdout:[mgr] 2024-09-17T23:03:59.555 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-09-17T23:03:59.555 INFO:teuthology.orchestra.run.smithi006.stdout: debug mgr = 20 2024-09-17T23:03:59.555 INFO:teuthology.orchestra.run.smithi006.stdout: debug mon = 20 2024-09-17T23:03:59.555 INFO:teuthology.orchestra.run.smithi006.stdout: debug auth = 20 2024-09-17T23:03:59.555 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min pgs per osd = 4 2024-09-17T23:03:59.555 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min bytes per osd = 10 2024-09-17T23:03:59.555 INFO:teuthology.orchestra.run.smithi006.stdout: mgr/telemetry/nag = false 2024-09-17T23:03:59.556 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:03:59.556 INFO:teuthology.orchestra.run.smithi006.stdout:[mon] 2024-09-17T23:03:59.556 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-09-17T23:03:59.556 INFO:teuthology.orchestra.run.smithi006.stdout: debug mon = 20 2024-09-17T23:03:59.556 INFO:teuthology.orchestra.run.smithi006.stdout: debug paxos = 20 2024-09-17T23:03:59.556 INFO:teuthology.orchestra.run.smithi006.stdout: debug auth = 20 2024-09-17T23:03:59.556 INFO:teuthology.orchestra.run.smithi006.stdout: mon data avail warn = 5 2024-09-17T23:03:59.556 INFO:teuthology.orchestra.run.smithi006.stdout: mon mgr mkfs grace = 240 2024-09-17T23:03:59.556 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min pgs per osd = 4 2024-09-17T23:03:59.556 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd reporter subtree level = osd 2024-09-17T23:03:59.556 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd prime pg temp = true 2024-09-17T23:03:59.556 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min bytes per osd = 10 2024-09-17T23:03:59.556 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:03:59.556 INFO:teuthology.orchestra.run.smithi006.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-17T23:03:59.556 INFO:teuthology.orchestra.run.smithi006.stdout: auth mon ticket ttl = 660 # 11m 2024-09-17T23:03:59.557 INFO:teuthology.orchestra.run.smithi006.stdout: auth service ticket ttl = 240 # 4m 2024-09-17T23:03:59.557 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:03:59.557 INFO:teuthology.orchestra.run.smithi006.stdout: # don't complain about insecure global_id in the test suite 2024-09-17T23:03:59.557 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-17T23:03:59.557 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-17T23:03:59.557 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:03:59.557 INFO:teuthology.orchestra.run.smithi006.stdout: # 1m isn't quite enough 2024-09-17T23:03:59.557 INFO:teuthology.orchestra.run.smithi006.stdout: mon_down_mkfs_grace = 2m 2024-09-17T23:03:59.557 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:03:59.557 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_filestore_osds = false 2024-09-17T23:03:59.557 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:03:59.557 INFO:teuthology.orchestra.run.smithi006.stdout:[client] 2024-09-17T23:03:59.557 INFO:teuthology.orchestra.run.smithi006.stdout: rgw cache enabled = true 2024-09-17T23:03:59.557 INFO:teuthology.orchestra.run.smithi006.stdout: rgw enable ops log = true 2024-09-17T23:03:59.557 INFO:teuthology.orchestra.run.smithi006.stdout: rgw enable usage log = true 2024-09-17T23:03:59.558 INFO:teuthology.orchestra.run.smithi006.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-17T23:03:59.558 INFO:teuthology.orchestra.run.smithi006.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-17T23:03:59.558 INFO:teuthology.orchestra.run.smithi006.stdout: debug rbd = 20 2024-09-17T23:03:59.558 INFO:teuthology.orchestra.run.smithi006.stdout: debug rbd_mirror = 15 2024-09-17T23:03:59.558 INFO:teuthology.orchestra.run.smithi006.stdout: log to stderr = False 2024-09-17T23:03:59.558 INFO:teuthology.orchestra.run.smithi006.stdout: rbd default features = 125 2024-09-17T23:03:59.558 INFO:teuthology.orchestra.run.smithi006.stdout: rbd mirror image policy type = none 2024-09-17T23:03:59.558 INFO:teuthology.orchestra.run.smithi006.stdout:[mon.a] 2024-09-17T23:03:59.558 INFO:teuthology.orchestra.run.smithi006.stdout:[client.mirror.0] 2024-09-17T23:03:59.558 INFO:teuthology.orchestra.run.smithi006.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-17T23:03:59.558 INFO:teuthology.orchestra.run.smithi006.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-17T23:03:59.558 INFO:teuthology.orchestra.run.smithi006.stdout:[client.mirror.1] 2024-09-17T23:03:59.558 INFO:teuthology.orchestra.run.smithi006.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-17T23:03:59.558 INFO:teuthology.orchestra.run.smithi006.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-17T23:03:59.558 INFO:teuthology.orchestra.run.smithi006.stdout:[client.mirror.2] 2024-09-17T23:03:59.558 INFO:teuthology.orchestra.run.smithi006.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-17T23:03:59.559 INFO:teuthology.orchestra.run.smithi006.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-17T23:03:59.559 INFO:teuthology.orchestra.run.smithi006.stdout:[client.mirror.3] 2024-09-17T23:03:59.559 INFO:teuthology.orchestra.run.smithi006.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-17T23:03:59.559 INFO:teuthology.orchestra.run.smithi006.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-17T23:03:59.559 INFO:teuthology.orchestra.run.smithi038.stdout:[global] 2024-09-17T23:03:59.559 INFO:teuthology.orchestra.run.smithi038.stdout: chdir = "" 2024-09-17T23:03:59.559 INFO:teuthology.orchestra.run.smithi038.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-17T23:03:59.559 INFO:teuthology.orchestra.run.smithi038.stdout: auth supported = cephx 2024-09-17T23:03:59.560 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:03:59.560 INFO:teuthology.orchestra.run.smithi038.stdout: filestore xattr use omap = true 2024-09-17T23:03:59.560 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:03:59.560 INFO:teuthology.orchestra.run.smithi038.stdout: mon clock drift allowed = 1.000 2024-09-17T23:03:59.560 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:03:59.560 INFO:teuthology.orchestra.run.smithi038.stdout: osd crush chooseleaf type = 0 2024-09-17T23:03:59.560 INFO:teuthology.orchestra.run.smithi038.stdout: auth debug = true 2024-09-17T23:03:59.560 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:03:59.560 INFO:teuthology.orchestra.run.smithi038.stdout: ms die on old message = true 2024-09-17T23:03:59.560 INFO:teuthology.orchestra.run.smithi038.stdout: ms die on bug = true 2024-09-17T23:03:59.560 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:03:59.560 INFO:teuthology.orchestra.run.smithi038.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-17T23:03:59.560 INFO:teuthology.orchestra.run.smithi038.stdout: mon pg warn max object skew = 0 2024-09-17T23:03:59.561 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:03:59.561 INFO:teuthology.orchestra.run.smithi038.stdout: # disable pg_autoscaler by default for new pools 2024-09-17T23:03:59.561 INFO:teuthology.orchestra.run.smithi038.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-17T23:03:59.561 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:03:59.561 INFO:teuthology.orchestra.run.smithi038.stdout: osd pool default size = 2 2024-09-17T23:03:59.561 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:03:59.561 INFO:teuthology.orchestra.run.smithi038.stdout: mon osd allow primary affinity = true 2024-09-17T23:03:59.561 INFO:teuthology.orchestra.run.smithi038.stdout: mon osd allow pg remap = true 2024-09-17T23:03:59.561 INFO:teuthology.orchestra.run.smithi038.stdout: mon warn on legacy crush tunables = false 2024-09-17T23:03:59.561 INFO:teuthology.orchestra.run.smithi038.stdout: mon warn on crush straw calc version zero = false 2024-09-17T23:03:59.561 INFO:teuthology.orchestra.run.smithi038.stdout: mon warn on no sortbitwise = false 2024-09-17T23:03:59.561 INFO:teuthology.orchestra.run.smithi038.stdout: mon warn on osd down out interval zero = false 2024-09-17T23:03:59.561 INFO:teuthology.orchestra.run.smithi038.stdout: mon warn on too few osds = false 2024-09-17T23:03:59.561 INFO:teuthology.orchestra.run.smithi038.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-17T23:03:59.562 INFO:teuthology.orchestra.run.smithi038.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-17T23:03:59.562 INFO:teuthology.orchestra.run.smithi038.stdout: mon_allow_pool_size_one = true 2024-09-17T23:03:59.562 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:03:59.562 INFO:teuthology.orchestra.run.smithi038.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-17T23:03:59.562 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:03:59.562 INFO:teuthology.orchestra.run.smithi038.stdout: osd default data pool replay window = 5 2024-09-17T23:03:59.562 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:03:59.562 INFO:teuthology.orchestra.run.smithi038.stdout: mon allow pool delete = true 2024-09-17T23:03:59.562 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:03:59.562 INFO:teuthology.orchestra.run.smithi038.stdout: mon cluster log file level = debug 2024-09-17T23:03:59.562 INFO:teuthology.orchestra.run.smithi038.stdout: debug asserts on shutdown = true 2024-09-17T23:03:59.562 INFO:teuthology.orchestra.run.smithi038.stdout: mon health detail to clog = false 2024-09-17T23:03:59.562 INFO:teuthology.orchestra.run.smithi038.stdout: mon host = 172.21.15.6 2024-09-17T23:03:59.562 INFO:teuthology.orchestra.run.smithi038.stdout: mon client directed command retry = 5 2024-09-17T23:03:59.563 INFO:teuthology.orchestra.run.smithi038.stdout: mon warn on pool no app = False 2024-09-17T23:03:59.563 INFO:teuthology.orchestra.run.smithi038.stdout: ms inject socket failures = 5000 2024-09-17T23:03:59.563 INFO:teuthology.orchestra.run.smithi038.stdout: fsid = 545dc37d-b29c-4f04-aad4-46458bcf9f65 2024-09-17T23:03:59.563 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:03:59.563 INFO:teuthology.orchestra.run.smithi038.stdout:[osd] 2024-09-17T23:03:59.563 INFO:teuthology.orchestra.run.smithi038.stdout: osd journal size = 100 2024-09-17T23:03:59.563 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:03:59.563 INFO:teuthology.orchestra.run.smithi038.stdout: osd scrub load threshold = 5.0 2024-09-17T23:03:59.563 INFO:teuthology.orchestra.run.smithi038.stdout: osd scrub max interval = 600 2024-09-17T23:03:59.563 INFO:teuthology.orchestra.run.smithi038.stdout: osd mclock profile = high_recovery_ops 2024-09-17T23:03:59.563 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:03:59.563 INFO:teuthology.orchestra.run.smithi038.stdout: osd recover clone overlap = true 2024-09-17T23:03:59.563 INFO:teuthology.orchestra.run.smithi038.stdout: osd recovery max chunk = 1048576 2024-09-17T23:03:59.563 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:03:59.564 INFO:teuthology.orchestra.run.smithi038.stdout: osd debug shutdown = true 2024-09-17T23:03:59.564 INFO:teuthology.orchestra.run.smithi038.stdout: osd debug op order = true 2024-09-17T23:03:59.564 INFO:teuthology.orchestra.run.smithi038.stdout: osd debug verify stray on activate = true 2024-09-17T23:03:59.564 INFO:teuthology.orchestra.run.smithi038.stdout: osd debug trim objects = true 2024-09-17T23:03:59.564 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:03:59.564 INFO:teuthology.orchestra.run.smithi038.stdout: osd open classes on start = true 2024-09-17T23:03:59.564 INFO:teuthology.orchestra.run.smithi038.stdout: osd debug pg log writeout = true 2024-09-17T23:03:59.564 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:03:59.564 INFO:teuthology.orchestra.run.smithi038.stdout: osd deep scrub update digest min age = 30 2024-09-17T23:03:59.564 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:03:59.564 INFO:teuthology.orchestra.run.smithi038.stdout: osd map max advance = 10 2024-09-17T23:03:59.564 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:03:59.564 INFO:teuthology.orchestra.run.smithi038.stdout: journal zero on create = true 2024-09-17T23:03:59.564 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:03:59.564 INFO:teuthology.orchestra.run.smithi038.stdout: filestore ondisk finisher threads = 3 2024-09-17T23:03:59.565 INFO:teuthology.orchestra.run.smithi038.stdout: filestore apply finisher threads = 3 2024-09-17T23:03:59.565 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:03:59.565 INFO:teuthology.orchestra.run.smithi038.stdout: bdev debug aio = true 2024-09-17T23:03:59.565 INFO:teuthology.orchestra.run.smithi038.stdout: osd debug misdirected ops = true 2024-09-17T23:03:59.565 INFO:teuthology.orchestra.run.smithi038.stdout: bluestore block size = 96636764160 2024-09-17T23:03:59.565 INFO:teuthology.orchestra.run.smithi038.stdout: bluestore compression algorithm = zstd 2024-09-17T23:03:59.565 INFO:teuthology.orchestra.run.smithi038.stdout: bluestore compression mode = aggressive 2024-09-17T23:03:59.565 INFO:teuthology.orchestra.run.smithi038.stdout: bluestore fsck on mount = True 2024-09-17T23:03:59.565 INFO:teuthology.orchestra.run.smithi038.stdout: debug bluefs = 1/20 2024-09-17T23:03:59.565 INFO:teuthology.orchestra.run.smithi038.stdout: debug bluestore = 1/20 2024-09-17T23:03:59.565 INFO:teuthology.orchestra.run.smithi038.stdout: debug ms = 1 2024-09-17T23:03:59.565 INFO:teuthology.orchestra.run.smithi038.stdout: debug osd = 20 2024-09-17T23:03:59.565 INFO:teuthology.orchestra.run.smithi038.stdout: debug rocksdb = 4/10 2024-09-17T23:03:59.565 INFO:teuthology.orchestra.run.smithi038.stdout: mon osd backfillfull_ratio = 0.85 2024-09-17T23:03:59.565 INFO:teuthology.orchestra.run.smithi038.stdout: mon osd full ratio = 0.9 2024-09-17T23:03:59.566 INFO:teuthology.orchestra.run.smithi038.stdout: mon osd nearfull ratio = 0.8 2024-09-17T23:03:59.566 INFO:teuthology.orchestra.run.smithi038.stdout: osd failsafe full ratio = 0.95 2024-09-17T23:03:59.566 INFO:teuthology.orchestra.run.smithi038.stdout: osd objectstore = bluestore 2024-09-17T23:03:59.566 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:03:59.566 INFO:teuthology.orchestra.run.smithi038.stdout:[mgr] 2024-09-17T23:03:59.566 INFO:teuthology.orchestra.run.smithi038.stdout: debug ms = 1 2024-09-17T23:03:59.566 INFO:teuthology.orchestra.run.smithi038.stdout: debug mgr = 20 2024-09-17T23:03:59.566 INFO:teuthology.orchestra.run.smithi038.stdout: debug mon = 20 2024-09-17T23:03:59.566 INFO:teuthology.orchestra.run.smithi038.stdout: debug auth = 20 2024-09-17T23:03:59.566 INFO:teuthology.orchestra.run.smithi038.stdout: mon reweight min pgs per osd = 4 2024-09-17T23:03:59.566 INFO:teuthology.orchestra.run.smithi038.stdout: mon reweight min bytes per osd = 10 2024-09-17T23:03:59.566 INFO:teuthology.orchestra.run.smithi038.stdout: mgr/telemetry/nag = false 2024-09-17T23:03:59.566 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:03:59.566 INFO:teuthology.orchestra.run.smithi038.stdout:[mon] 2024-09-17T23:03:59.566 INFO:teuthology.orchestra.run.smithi038.stdout: debug ms = 1 2024-09-17T23:03:59.567 INFO:teuthology.orchestra.run.smithi038.stdout: debug mon = 20 2024-09-17T23:03:59.567 INFO:teuthology.orchestra.run.smithi038.stdout: debug paxos = 20 2024-09-17T23:03:59.567 INFO:teuthology.orchestra.run.smithi038.stdout: debug auth = 20 2024-09-17T23:03:59.567 INFO:teuthology.orchestra.run.smithi038.stdout: mon data avail warn = 5 2024-09-17T23:03:59.567 INFO:teuthology.orchestra.run.smithi038.stdout: mon mgr mkfs grace = 240 2024-09-17T23:03:59.567 INFO:teuthology.orchestra.run.smithi038.stdout: mon reweight min pgs per osd = 4 2024-09-17T23:03:59.567 INFO:teuthology.orchestra.run.smithi038.stdout: mon osd reporter subtree level = osd 2024-09-17T23:03:59.567 INFO:teuthology.orchestra.run.smithi038.stdout: mon osd prime pg temp = true 2024-09-17T23:03:59.567 INFO:teuthology.orchestra.run.smithi038.stdout: mon reweight min bytes per osd = 10 2024-09-17T23:03:59.567 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:03:59.567 INFO:teuthology.orchestra.run.smithi038.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-17T23:03:59.567 INFO:teuthology.orchestra.run.smithi038.stdout: auth mon ticket ttl = 660 # 11m 2024-09-17T23:03:59.567 INFO:teuthology.orchestra.run.smithi038.stdout: auth service ticket ttl = 240 # 4m 2024-09-17T23:03:59.567 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:03:59.568 INFO:teuthology.orchestra.run.smithi038.stdout: # don't complain about insecure global_id in the test suite 2024-09-17T23:03:59.568 INFO:teuthology.orchestra.run.smithi038.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-17T23:03:59.568 INFO:teuthology.orchestra.run.smithi038.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-17T23:03:59.568 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:03:59.568 INFO:teuthology.orchestra.run.smithi038.stdout: # 1m isn't quite enough 2024-09-17T23:03:59.568 INFO:teuthology.orchestra.run.smithi038.stdout: mon_down_mkfs_grace = 2m 2024-09-17T23:03:59.568 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:03:59.568 INFO:teuthology.orchestra.run.smithi038.stdout: mon_warn_on_filestore_osds = false 2024-09-17T23:03:59.568 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:03:59.568 INFO:teuthology.orchestra.run.smithi038.stdout:[client] 2024-09-17T23:03:59.568 INFO:teuthology.orchestra.run.smithi038.stdout: rgw cache enabled = true 2024-09-17T23:03:59.568 INFO:teuthology.orchestra.run.smithi038.stdout: rgw enable ops log = true 2024-09-17T23:03:59.568 INFO:teuthology.orchestra.run.smithi038.stdout: rgw enable usage log = true 2024-09-17T23:03:59.568 INFO:teuthology.orchestra.run.smithi038.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-17T23:03:59.568 INFO:teuthology.orchestra.run.smithi038.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-17T23:03:59.569 INFO:teuthology.orchestra.run.smithi038.stdout: debug rbd = 20 2024-09-17T23:03:59.569 INFO:teuthology.orchestra.run.smithi038.stdout: debug rbd_mirror = 15 2024-09-17T23:03:59.569 INFO:teuthology.orchestra.run.smithi038.stdout: log to stderr = False 2024-09-17T23:03:59.569 INFO:teuthology.orchestra.run.smithi038.stdout: rbd default features = 125 2024-09-17T23:03:59.569 INFO:teuthology.orchestra.run.smithi038.stdout: rbd mirror image policy type = none 2024-09-17T23:03:59.569 INFO:teuthology.orchestra.run.smithi038.stdout:[mon.a] 2024-09-17T23:03:59.569 INFO:teuthology.orchestra.run.smithi038.stdout:[client.mirror.0] 2024-09-17T23:03:59.569 INFO:teuthology.orchestra.run.smithi038.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-17T23:03:59.569 INFO:teuthology.orchestra.run.smithi038.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-17T23:03:59.569 INFO:teuthology.orchestra.run.smithi038.stdout:[client.mirror.1] 2024-09-17T23:03:59.569 INFO:teuthology.orchestra.run.smithi038.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-17T23:03:59.569 INFO:teuthology.orchestra.run.smithi038.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-17T23:03:59.569 INFO:teuthology.orchestra.run.smithi038.stdout:[client.mirror.2] 2024-09-17T23:03:59.569 INFO:teuthology.orchestra.run.smithi038.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-17T23:03:59.570 INFO:teuthology.orchestra.run.smithi038.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-17T23:03:59.570 INFO:teuthology.orchestra.run.smithi038.stdout:[client.mirror.3] 2024-09-17T23:03:59.570 INFO:teuthology.orchestra.run.smithi038.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-17T23:03:59.570 INFO:teuthology.orchestra.run.smithi038.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-17T23:03:59.574 INFO:tasks.ceph:Creating admin key on cluster1.mon.a... 2024-09-17T23:03:59.574 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/cluster1.keyring 2024-09-17T23:03:59.670 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-17T23:03:59.670 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-09-17T23:03:59.670 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/etc/ceph/cluster1.keyring of=/dev/stdout 2024-09-17T23:03:59.686 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-09-17T23:03:59.686 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/home/ubuntu/cephtest/cluster1.monmap of=/dev/stdout 2024-09-17T23:03:59.742 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi006.front.sepia.ceph.com 2024-09-17T23:03:59.742 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-09-17T23:03:59.742 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/etc/ceph/cluster1.keyring 2024-09-17T23:03:59.742 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2024-09-17T23:03:59.818 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-09-17T23:03:59.818 DEBUG:teuthology.orchestra.run.smithi006:> dd of=/home/ubuntu/cephtest/cluster1.monmap 2024-09-17T23:03:59.874 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi038.front.sepia.ceph.com 2024-09-17T23:03:59.874 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-09-17T23:03:59.874 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/etc/ceph/cluster1.keyring 2024-09-17T23:03:59.874 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2024-09-17T23:03:59.912 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-09-17T23:03:59.912 DEBUG:teuthology.orchestra.run.smithi038:> dd of=/home/ubuntu/cephtest/cluster1.monmap 2024-09-17T23:03:59.969 INFO:tasks.ceph:Setting up mon nodes... 2024-09-17T23:03:59.969 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-17T23:03:59.969 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /var/lib/ceph/mgr/cluster1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/cluster1-x/keyring 2024-09-17T23:04:00.021 INFO:teuthology.orchestra.run.smithi006.stdout:creating /var/lib/ceph/mgr/cluster1-x/keyring 2024-09-17T23:04:00.024 INFO:tasks.ceph:Setting up mds nodes... 2024-09-17T23:04:00.024 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-17T23:04:00.024 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/cluster1.client.0.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.0.keyring 2024-09-17T23:04:00.066 INFO:teuthology.orchestra.run.smithi006.stdout:creating /etc/ceph/cluster1.client.0.keyring 2024-09-17T23:04:00.079 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror /etc/ceph/cluster1.client.mirror.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.keyring 2024-09-17T23:04:00.131 INFO:teuthology.orchestra.run.smithi038.stdout:creating /etc/ceph/cluster1.client.mirror.keyring 2024-09-17T23:04:00.145 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.0 /etc/ceph/cluster1.client.mirror.0.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.0.keyring 2024-09-17T23:04:00.188 INFO:teuthology.orchestra.run.smithi038.stdout:creating /etc/ceph/cluster1.client.mirror.0.keyring 2024-09-17T23:04:00.201 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.1 /etc/ceph/cluster1.client.mirror.1.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.1.keyring 2024-09-17T23:04:00.283 INFO:teuthology.orchestra.run.smithi038.stdout:creating /etc/ceph/cluster1.client.mirror.1.keyring 2024-09-17T23:04:00.296 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.2 /etc/ceph/cluster1.client.mirror.2.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.2.keyring 2024-09-17T23:04:00.378 INFO:teuthology.orchestra.run.smithi038.stdout:creating /etc/ceph/cluster1.client.mirror.2.keyring 2024-09-17T23:04:00.391 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.3 /etc/ceph/cluster1.client.mirror.3.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.3.keyring 2024-09-17T23:04:00.474 INFO:teuthology.orchestra.run.smithi038.stdout:creating /etc/ceph/cluster1.client.mirror.3.keyring 2024-09-17T23:04:00.487 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.4 /etc/ceph/cluster1.client.mirror.4.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.4.keyring 2024-09-17T23:04:00.569 INFO:teuthology.orchestra.run.smithi038.stdout:creating /etc/ceph/cluster1.client.mirror.4.keyring 2024-09-17T23:04:00.583 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.5 /etc/ceph/cluster1.client.mirror.5.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.5.keyring 2024-09-17T23:04:00.665 INFO:teuthology.orchestra.run.smithi038.stdout:creating /etc/ceph/cluster1.client.mirror.5.keyring 2024-09-17T23:04:00.678 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.6 /etc/ceph/cluster1.client.mirror.6.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.6.keyring 2024-09-17T23:04:00.761 INFO:teuthology.orchestra.run.smithi038.stdout:creating /etc/ceph/cluster1.client.mirror.6.keyring 2024-09-17T23:04:00.774 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-17T23:04:00.774 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi006.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-17T23:04:00.775 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /var/lib/ceph/osd/cluster1-0 2024-09-17T23:04:00.802 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-17T23:04:00.802 INFO:tasks.ceph:role: cluster1.osd.0 2024-09-17T23:04:00.803 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi006.front.sepia.ceph.com 2024-09-17T23:04:00.803 DEBUG:teuthology.orchestra.run.smithi006:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-17T23:04:00.877 INFO:teuthology.orchestra.run.smithi006.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-17T23:04:00.877 INFO:teuthology.orchestra.run.smithi006.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-17T23:04:00.877 INFO:teuthology.orchestra.run.smithi006.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-17T23:04:00.877 INFO:teuthology.orchestra.run.smithi006.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-17T23:04:00.877 INFO:teuthology.orchestra.run.smithi006.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-17T23:04:00.877 INFO:teuthology.orchestra.run.smithi006.stdout: = sunit=0 swidth=0 blks 2024-09-17T23:04:00.877 INFO:teuthology.orchestra.run.smithi006.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-17T23:04:00.877 INFO:teuthology.orchestra.run.smithi006.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-17T23:04:00.878 INFO:teuthology.orchestra.run.smithi006.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-17T23:04:00.878 INFO:teuthology.orchestra.run.smithi006.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-17T23:04:01.376 INFO:teuthology.orchestra.run.smithi006.stdout:Discarding blocks...Done. 2024-09-17T23:04:01.378 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi006.front.sepia.ceph.com -o noatime 2024-09-17T23:04:01.378 DEBUG:teuthology.orchestra.run.smithi006:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/cluster1-0 2024-09-17T23:04:01.523 DEBUG:teuthology.orchestra.run.smithi006:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-0 2024-09-17T23:04:01.560 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /var/lib/ceph/osd/cluster1-1 2024-09-17T23:04:01.629 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-17T23:04:01.629 INFO:tasks.ceph:role: cluster1.osd.1 2024-09-17T23:04:01.629 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi006.front.sepia.ceph.com 2024-09-17T23:04:01.629 DEBUG:teuthology.orchestra.run.smithi006:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-17T23:04:01.704 INFO:teuthology.orchestra.run.smithi006.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-17T23:04:01.704 INFO:teuthology.orchestra.run.smithi006.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-17T23:04:01.704 INFO:teuthology.orchestra.run.smithi006.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-17T23:04:01.704 INFO:teuthology.orchestra.run.smithi006.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-17T23:04:01.704 INFO:teuthology.orchestra.run.smithi006.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-17T23:04:01.704 INFO:teuthology.orchestra.run.smithi006.stdout: = sunit=0 swidth=0 blks 2024-09-17T23:04:01.705 INFO:teuthology.orchestra.run.smithi006.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-17T23:04:01.705 INFO:teuthology.orchestra.run.smithi006.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-17T23:04:01.705 INFO:teuthology.orchestra.run.smithi006.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-17T23:04:01.705 INFO:teuthology.orchestra.run.smithi006.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-17T23:04:02.144 INFO:teuthology.orchestra.run.smithi006.stdout:Discarding blocks...Done. 2024-09-17T23:04:02.146 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi006.front.sepia.ceph.com -o noatime 2024-09-17T23:04:02.147 DEBUG:teuthology.orchestra.run.smithi006:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/cluster1-1 2024-09-17T23:04:02.236 DEBUG:teuthology.orchestra.run.smithi006:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-1 2024-09-17T23:04:02.265 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /var/lib/ceph/osd/cluster1-2 2024-09-17T23:04:02.332 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-17T23:04:02.332 INFO:tasks.ceph:role: cluster1.osd.2 2024-09-17T23:04:02.333 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi006.front.sepia.ceph.com 2024-09-17T23:04:02.333 DEBUG:teuthology.orchestra.run.smithi006:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-17T23:04:02.405 INFO:teuthology.orchestra.run.smithi006.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-17T23:04:02.405 INFO:teuthology.orchestra.run.smithi006.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-17T23:04:02.405 INFO:teuthology.orchestra.run.smithi006.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-17T23:04:02.405 INFO:teuthology.orchestra.run.smithi006.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-17T23:04:02.405 INFO:teuthology.orchestra.run.smithi006.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-17T23:04:02.405 INFO:teuthology.orchestra.run.smithi006.stdout: = sunit=0 swidth=0 blks 2024-09-17T23:04:02.405 INFO:teuthology.orchestra.run.smithi006.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-17T23:04:02.405 INFO:teuthology.orchestra.run.smithi006.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-17T23:04:02.405 INFO:teuthology.orchestra.run.smithi006.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-17T23:04:02.405 INFO:teuthology.orchestra.run.smithi006.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-17T23:04:02.884 INFO:teuthology.orchestra.run.smithi006.stdout:Discarding blocks...Done. 2024-09-17T23:04:02.886 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi006.front.sepia.ceph.com -o noatime 2024-09-17T23:04:02.886 DEBUG:teuthology.orchestra.run.smithi006:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/cluster1-2 2024-09-17T23:04:03.031 DEBUG:teuthology.orchestra.run.smithi006:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-2 2024-09-17T23:04:03.061 DEBUG:teuthology.orchestra.run.smithi006:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/cluster1.monmap 2024-09-17T23:04:03.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-17T23:04:03.152+0000 7f209ce1d640 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-0/keyring: can't open /var/lib/ceph/osd/cluster1-0/keyring: (2) No such file or directory 2024-09-17T23:04:03.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-17T23:04:03.154+0000 7f209ce1d640 -1 created new key in keyring /var/lib/ceph/osd/cluster1-0/keyring 2024-09-17T23:04:03.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-17T23:04:03.154+0000 7f209ce1d640 -1 bdev(0x55fc8c43d800 /var/lib/ceph/osd/cluster1-0/block) open stat got: (1) Operation not permitted 2024-09-17T23:04:03.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-17T23:04:03.154+0000 7f209ce1d640 -1 bluestore(/var/lib/ceph/osd/cluster1-0) _read_fsid unparsable uuid 2024-09-17T23:04:06.331 DEBUG:teuthology.orchestra.run.smithi006:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-0 2024-09-17T23:04:06.360 DEBUG:teuthology.orchestra.run.smithi006:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/cluster1.monmap 2024-09-17T23:04:06.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-17T23:04:06.446+0000 7f917040f640 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-1/keyring: can't open /var/lib/ceph/osd/cluster1-1/keyring: (2) No such file or directory 2024-09-17T23:04:06.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-17T23:04:06.447+0000 7f917040f640 -1 created new key in keyring /var/lib/ceph/osd/cluster1-1/keyring 2024-09-17T23:04:06.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-17T23:04:06.447+0000 7f917040f640 -1 bdev(0x55ac70461800 /var/lib/ceph/osd/cluster1-1/block) open stat got: (1) Operation not permitted 2024-09-17T23:04:06.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-17T23:04:06.447+0000 7f917040f640 -1 bluestore(/var/lib/ceph/osd/cluster1-1) _read_fsid unparsable uuid 2024-09-17T23:04:09.853 DEBUG:teuthology.orchestra.run.smithi006:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-1 2024-09-17T23:04:09.883 DEBUG:teuthology.orchestra.run.smithi006:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/cluster1.monmap 2024-09-17T23:04:09.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-17T23:04:09.973+0000 7f9b4e611640 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-2/keyring: can't open /var/lib/ceph/osd/cluster1-2/keyring: (2) No such file or directory 2024-09-17T23:04:09.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-17T23:04:09.974+0000 7f9b4e611640 -1 created new key in keyring /var/lib/ceph/osd/cluster1-2/keyring 2024-09-17T23:04:09.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-17T23:04:09.974+0000 7f9b4e611640 -1 bdev(0x560480259800 /var/lib/ceph/osd/cluster1-2/block) open stat got: (1) Operation not permitted 2024-09-17T23:04:09.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-17T23:04:09.974+0000 7f9b4e611640 -1 bluestore(/var/lib/ceph/osd/cluster1-2) _read_fsid unparsable uuid 2024-09-17T23:04:13.127 DEBUG:teuthology.orchestra.run.smithi006:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-2 2024-09-17T23:04:13.155 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-17T23:04:13.155 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-09-17T23:04:13.156 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/mgr/cluster1-x/keyring of=/dev/stdout 2024-09-17T23:04:13.223 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-09-17T23:04:13.224 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/osd/cluster1-0/keyring of=/dev/stdout 2024-09-17T23:04:13.291 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-09-17T23:04:13.291 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/osd/cluster1-1/keyring of=/dev/stdout 2024-09-17T23:04:13.358 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-09-17T23:04:13.358 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/osd/cluster1-2/keyring of=/dev/stdout 2024-09-17T23:04:13.425 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-09-17T23:04:13.425 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/etc/ceph/cluster1.client.0.keyring of=/dev/stdout 2024-09-17T23:04:13.482 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-09-17T23:04:13.482 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/etc/ceph/cluster1.client.mirror.keyring of=/dev/stdout 2024-09-17T23:04:13.502 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-09-17T23:04:13.502 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/etc/ceph/cluster1.client.mirror.0.keyring of=/dev/stdout 2024-09-17T23:04:13.561 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-09-17T23:04:13.561 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/etc/ceph/cluster1.client.mirror.1.keyring of=/dev/stdout 2024-09-17T23:04:13.619 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-09-17T23:04:13.619 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/etc/ceph/cluster1.client.mirror.2.keyring of=/dev/stdout 2024-09-17T23:04:13.677 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-09-17T23:04:13.677 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/etc/ceph/cluster1.client.mirror.3.keyring of=/dev/stdout 2024-09-17T23:04:13.734 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-09-17T23:04:13.735 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/etc/ceph/cluster1.client.mirror.4.keyring of=/dev/stdout 2024-09-17T23:04:13.792 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-09-17T23:04:13.792 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/etc/ceph/cluster1.client.mirror.5.keyring of=/dev/stdout 2024-09-17T23:04:13.850 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-09-17T23:04:13.850 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/etc/ceph/cluster1.client.mirror.6.keyring of=/dev/stdout 2024-09-17T23:04:13.907 INFO:tasks.ceph:Adding keys to all mons... 2024-09-17T23:04:13.908 DEBUG:teuthology.orchestra.run.smithi006:> sudo tee -a /etc/ceph/cluster1.keyring 2024-09-17T23:04:13.934 INFO:teuthology.orchestra.run.smithi006.stdout:[mgr.x] 2024-09-17T23:04:13.934 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQDgCupmG8E0ARAAsQjqIXSRHUUGypt1LBQ66Q== 2024-09-17T23:04:13.934 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.0] 2024-09-17T23:04:13.935 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQDjCupmb70zCRAAal6o+ANJE72YuLgkhM0yjQ== 2024-09-17T23:04:13.935 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.1] 2024-09-17T23:04:13.935 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQDmCupmfmm+GhAAZLQyOQvlO9Q+pfZkAraE/A== 2024-09-17T23:04:13.935 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.2] 2024-09-17T23:04:13.935 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQDpCupmZe0pOhAAZ3Mp3dMrBIt8DsrmxPvqHQ== 2024-09-17T23:04:13.935 INFO:teuthology.orchestra.run.smithi006.stdout:[client.0] 2024-09-17T23:04:13.935 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQDgCupmWjLgAxAAYQsiTvPqyQFtLrS/9vPzoA== 2024-09-17T23:04:13.935 INFO:teuthology.orchestra.run.smithi006.stdout:[client.mirror] 2024-09-17T23:04:13.935 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQDgCupmMGrABxAA0J818ImLp/GgA5cQRJYz2g== 2024-09-17T23:04:13.935 INFO:teuthology.orchestra.run.smithi006.stdout:[client.mirror.0] 2024-09-17T23:04:13.935 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQDgCupmWjAgCxAAefibdjhobV0WoLTqfqk4lQ== 2024-09-17T23:04:13.935 INFO:teuthology.orchestra.run.smithi006.stdout:[client.mirror.1] 2024-09-17T23:04:13.935 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQDgCupm7h7VEBAArkVJ99C+P3IZ05GL9e/Fdw== 2024-09-17T23:04:13.935 INFO:teuthology.orchestra.run.smithi006.stdout:[client.mirror.2] 2024-09-17T23:04:13.936 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQDgCupmobB6FhAA7cvicbHshpJh/r3Io3aUKw== 2024-09-17T23:04:13.936 INFO:teuthology.orchestra.run.smithi006.stdout:[client.mirror.3] 2024-09-17T23:04:13.936 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQDgCupmqFYqHBAAd8snKc43AiQ6clVSoQSsPw== 2024-09-17T23:04:13.936 INFO:teuthology.orchestra.run.smithi006.stdout:[client.mirror.4] 2024-09-17T23:04:13.936 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQDgCupmI/jiIRAA4JydEDGy2PBOk3Mib1TTzA== 2024-09-17T23:04:13.936 INFO:teuthology.orchestra.run.smithi006.stdout:[client.mirror.5] 2024-09-17T23:04:13.936 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQDgCupm+7GUJxAAn4OrQI7tAGceuctBUqlHww== 2024-09-17T23:04:13.936 INFO:teuthology.orchestra.run.smithi006.stdout:[client.mirror.6] 2024-09-17T23:04:13.936 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQDgCupmCaFLLRAATDkNORzJuISWbCSABIaQJA== 2024-09-17T23:04:13.937 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-17T23:04:14.034 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T23:04:14.089 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T23:04:14.141 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T23:04:14.193 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-17T23:04:14.248 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-17T23:04:14.303 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-17T23:04:14.358 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-17T23:04:14.413 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-17T23:04:14.468 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-17T23:04:14.523 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.4 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-17T23:04:14.577 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.5 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-17T23:04:14.632 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.6 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-17T23:04:14.687 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-17T23:04:14.687 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /var/lib/ceph/mon/cluster1-a 2024-09-17T23:04:14.712 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster cluster1 --mkfs -i a --monmap /home/ubuntu/cephtest/cluster1.monmap --keyring /etc/ceph/cluster1.keyring 2024-09-17T23:04:14.811 DEBUG:teuthology.orchestra.run.smithi006:> sudo chown -R ceph:ceph /var/lib/ceph/mon/cluster1-a 2024-09-17T23:04:14.836 DEBUG:teuthology.orchestra.run.smithi006:> rm -- /home/ubuntu/cephtest/cluster1.monmap 2024-09-17T23:04:14.893 INFO:tasks.ceph:Starting mon daemons in cluster cluster1... 2024-09-17T23:04:14.893 INFO:tasks.ceph.cluster1.mon.a:Restarting daemon 2024-09-17T23:04:14.893 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster cluster1 -i a 2024-09-17T23:04:14.935 INFO:tasks.ceph.cluster1.mon.a:Started 2024-09-17T23:04:14.935 INFO:tasks.ceph:Starting mgr daemons in cluster cluster1... 2024-09-17T23:04:14.936 INFO:tasks.ceph.cluster1.mgr.x:Restarting daemon 2024-09-17T23:04:14.936 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster cluster1 -i x 2024-09-17T23:04:14.937 INFO:tasks.ceph.cluster1.mgr.x:Started 2024-09-17T23:04:14.938 INFO:tasks.ceph:Setting crush tunables to default 2024-09-17T23:04:14.938 DEBUG:teuthology.orchestra.run.smithi006:> sudo ceph --cluster cluster1 osd crush tunables default 2024-09-17T23:04:15.114 INFO:teuthology.orchestra.run.smithi006.stderr:adjusted tunables profile to default 2024-09-17T23:04:15.125 INFO:tasks.ceph:check_enable_crimson: False 2024-09-17T23:04:15.125 INFO:tasks.ceph:Starting osd daemons in cluster cluster1... 2024-09-17T23:04:15.126 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-09-17T23:04:15.126 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/osd/cluster1-0/fsid of=/dev/stdout 2024-09-17T23:04:15.151 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-09-17T23:04:15.151 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/osd/cluster1-1/fsid of=/dev/stdout 2024-09-17T23:04:15.217 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-09-17T23:04:15.217 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/osd/cluster1-2/fsid of=/dev/stdout 2024-09-17T23:04:15.283 DEBUG:teuthology.orchestra.run.smithi006:> sudo ceph --cluster cluster1 osd new 530ec6e3-0be9-4615-9d5a-2fa97fce2539 0 2024-09-17T23:04:15.336 INFO:tasks.ceph.cluster1.mgr.x.smithi006.stderr:2024-09-17T23:04:15.333+0000 7f6540e0e040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-17T23:04:15.381 INFO:tasks.ceph.cluster1.mgr.x.smithi006.stderr:2024-09-17T23:04:15.377+0000 7f6540e0e040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-17T23:04:15.524 INFO:teuthology.orchestra.run.smithi006.stdout:0 2024-09-17T23:04:15.535 DEBUG:teuthology.orchestra.run.smithi006:> sudo ceph --cluster cluster1 osd new 1d1e25d4-5837-450d-aae1-a44b3233c9ab 1 2024-09-17T23:04:15.548 INFO:tasks.ceph.cluster1.mgr.x.smithi006.stderr:2024-09-17T23:04:15.545+0000 7f6540e0e040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-17T23:04:15.604 INFO:tasks.ceph.cluster1.mgr.x.smithi006.stderr:2024-09-17T23:04:15.601+0000 7f6540e0e040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-17T23:04:15.733 INFO:teuthology.orchestra.run.smithi006.stdout:1 2024-09-17T23:04:15.744 DEBUG:teuthology.orchestra.run.smithi006:> sudo ceph --cluster cluster1 osd new 9540a0fe-29b1-4de7-b617-a765d3f99fc5 2 2024-09-17T23:04:15.809 INFO:tasks.ceph.cluster1.mgr.x.smithi006.stderr:2024-09-17T23:04:15.806+0000 7f6540e0e040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-17T23:04:15.926 INFO:teuthology.orchestra.run.smithi006.stdout:2 2024-09-17T23:04:15.930 INFO:tasks.ceph.cluster1.mgr.x.smithi006.stderr:2024-09-17T23:04:15.926+0000 7f6540e0e040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-17T23:04:15.938 INFO:tasks.ceph.cluster1.osd.0:Restarting daemon 2024-09-17T23:04:15.938 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 0 2024-09-17T23:04:15.940 INFO:tasks.ceph.cluster1.osd.0:Started 2024-09-17T23:04:15.941 INFO:tasks.ceph.cluster1.osd.1:Restarting daemon 2024-09-17T23:04:15.941 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 1 2024-09-17T23:04:15.943 INFO:tasks.ceph.cluster1.osd.1:Started 2024-09-17T23:04:15.943 INFO:tasks.ceph.cluster1.osd.2:Restarting daemon 2024-09-17T23:04:15.943 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 2 2024-09-17T23:04:15.945 INFO:tasks.ceph.cluster1.osd.2:Started 2024-09-17T23:04:15.946 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-09-17T23:04:16.157 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:04:16.157 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":5,"fsid":"545dc37d-b29c-4f04-aad4-46458bcf9f65","created":"2024-09-17T23:04:15.024128+0000","modified":"2024-09-17T23:04:15.924722+0000","last_up_change":"0.000000","last_in_change":"2024-09-17T23:04:15.924722+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"530ec6e3-0be9-4615-9d5a-2fa97fce2539","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1d1e25d4-5837-450d-aae1-a44b3233c9ab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9540a0fe-29b1-4de7-b617-a765d3f99fc5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T23:04:16.169 INFO:tasks.ceph.ceph_manager.cluster1:[] 2024-09-17T23:04:16.169 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-17T23:04:16.301 INFO:tasks.ceph.cluster1.mgr.x.smithi006.stderr:2024-09-17T23:04:16.298+0000 7f6540e0e040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-17T23:04:16.360 INFO:tasks.ceph.cluster1.mgr.x.smithi006.stderr:2024-09-17T23:04:16.357+0000 7f6540e0e040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-17T23:04:16.471 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2024-09-17T23:04:16.521 INFO:tasks.ceph.cluster1.mgr.x.smithi006.stderr:2024-09-17T23:04:16.518+0000 7f6540e0e040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-17T23:04:16.768 INFO:teuthology.misc.health.smithi006.stderr:2024-09-17T23:04:16.765+0000 7f9c777fe640 0 --2- 172.21.15.6:0/1898519851 >> v2:172.21.15.6:3300/0 conn(0x7f9c78144400 0x7f9c7813d8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-17T23:04:16.835 INFO:teuthology.misc.health.smithi006.stdout: 2024-09-17T23:04:16.835 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":5,"fsid":"545dc37d-b29c-4f04-aad4-46458bcf9f65","created":"2024-09-17T23:04:15.024128+0000","modified":"2024-09-17T23:04:15.924722+0000","last_up_change":"0.000000","last_in_change":"2024-09-17T23:04:15.924722+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"530ec6e3-0be9-4615-9d5a-2fa97fce2539","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1d1e25d4-5837-450d-aae1-a44b3233c9ab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9540a0fe-29b1-4de7-b617-a765d3f99fc5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T23:04:16.845 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-09-17T23:04:16.924 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-17T23:04:16.921+0000 7f4725f58640 -1 Falling back to public interface 2024-09-17T23:04:16.932 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-17T23:04:16.928+0000 7f8f09d25640 -1 Falling back to public interface 2024-09-17T23:04:17.439 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-17T23:04:17.434+0000 7f03a580d640 -1 Falling back to public interface 2024-09-17T23:04:17.521 INFO:tasks.ceph.cluster1.mgr.x.smithi006.stderr:2024-09-17T23:04:17.518+0000 7f6540e0e040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-17T23:04:17.576 INFO:tasks.ceph.cluster1.mgr.x.smithi006.stderr:2024-09-17T23:04:17.573+0000 7f6540e0e040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-17T23:04:17.963 INFO:tasks.ceph.cluster1.mgr.x.smithi006.stderr:2024-09-17T23:04:17.960+0000 7f6540e0e040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-17T23:04:18.053 INFO:tasks.ceph.cluster1.mgr.x.smithi006.stderr:2024-09-17T23:04:18.050+0000 7f6540e0e040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-17T23:04:18.165 INFO:tasks.ceph.cluster1.mgr.x.smithi006.stderr:2024-09-17T23:04:18.162+0000 7f6540e0e040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-17T23:04:18.213 INFO:tasks.ceph.cluster1.mgr.x.smithi006.stderr:2024-09-17T23:04:18.210+0000 7f6540e0e040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-17T23:04:18.332 INFO:tasks.ceph.cluster1.mgr.x.smithi006.stderr:2024-09-17T23:04:18.330+0000 7f6540e0e040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-17T23:04:18.388 INFO:tasks.ceph.cluster1.mgr.x.smithi006.stderr:2024-09-17T23:04:18.386+0000 7f6540e0e040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-17T23:04:18.435 INFO:tasks.ceph.cluster1.mgr.x.smithi006.stderr:2024-09-17T23:04:18.433+0000 7f6540e0e040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-17T23:04:18.552 INFO:tasks.ceph.cluster1.mgr.x.smithi006.stderr:2024-09-17T23:04:18.550+0000 7f6540e0e040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-17T23:04:18.606 INFO:tasks.ceph.cluster1.mgr.x.smithi006.stderr:2024-09-17T23:04:18.603+0000 7f6540e0e040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-17T23:04:18.659 INFO:tasks.ceph.cluster1.mgr.x.smithi006.stderr:2024-09-17T23:04:18.657+0000 7f6540e0e040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-17T23:04:18.730 INFO:tasks.ceph.cluster1.mgr.x.smithi006.stderr:2024-09-17T23:04:18.728+0000 7f6540e0e040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-17T23:04:18.776 INFO:tasks.ceph.cluster1.mgr.x.smithi006.stderr:2024-09-17T23:04:18.774+0000 7f6540e0e040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-17T23:04:18.823 INFO:tasks.ceph.cluster1.mgr.x.smithi006.stderr:2024-09-17T23:04:18.820+0000 7f6540e0e040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-17T23:04:19.507 INFO:tasks.ceph.cluster1.mgr.x.smithi006.stderr:2024-09-17T23:04:19.505+0000 7f6540e0e040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-17T23:04:20.574 INFO:tasks.ceph.cluster1.mgr.x.smithi006.stderr:2024-09-17T23:04:20.572+0000 7f64e0af6640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-09-17T23:04:20.730 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-17T23:04:20.727+0000 7f4725f58640 -1 osd.1 0 log_to_monitors true 2024-09-17T23:04:20.962 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-17T23:04:20.959+0000 7f03a580d640 -1 osd.2 0 log_to_monitors true 2024-09-17T23:04:21.266 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-17T23:04:21.264+0000 7f8f09d25640 -1 osd.0 0 log_to_monitors true 2024-09-17T23:04:23.147 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2024-09-17T23:04:23.605 INFO:teuthology.misc.health.smithi006.stdout: 2024-09-17T23:04:23.605 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":7,"fsid":"545dc37d-b29c-4f04-aad4-46458bcf9f65","created":"2024-09-17T23:04:15.024128+0000","modified":"2024-09-17T23:04:22.572327+0000","last_up_change":"0.000000","last_in_change":"2024-09-17T23:04:15.924722+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"530ec6e3-0be9-4615-9d5a-2fa97fce2539","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1d1e25d4-5837-450d-aae1-a44b3233c9ab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9540a0fe-29b1-4de7-b617-a765d3f99fc5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T23:04:23.622 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-09-17T23:04:23.674 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-17T23:04:23.672+0000 7f03a124b640 -1 osd.2 0 waiting for initial osdmap 2024-09-17T23:04:23.690 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-17T23:04:23.688+0000 7f039906b640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T23:04:23.703 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-17T23:04:23.701+0000 7f8f05cb4640 -1 osd.0 0 waiting for initial osdmap 2024-09-17T23:04:23.720 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-17T23:04:23.718+0000 7f8efdad4640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T23:04:23.733 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-17T23:04:23.731+0000 7f47226fb640 -1 osd.1 0 waiting for initial osdmap 2024-09-17T23:04:23.744 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-17T23:04:23.742+0000 7f4719d07640 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T23:04:29.925 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2024-09-17T23:04:30.221 INFO:teuthology.misc.health.smithi006.stdout: 2024-09-17T23:04:30.221 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":12,"fsid":"545dc37d-b29c-4f04-aad4-46458bcf9f65","created":"2024-09-17T23:04:15.024128+0000","modified":"2024-09-17T23:04:28.682721+0000","last_up_change":"2024-09-17T23:04:24.674100+0000","last_in_change":"2024-09-17T23:04:15.924722+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T23:04:25.756881+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"530ec6e3-0be9-4615-9d5a-2fa97fce2539","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":4294176576},{"type":"v1","addr":"172.21.15.6:6809","nonce":4294176576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":4294176576},{"type":"v1","addr":"172.21.15.6:6811","nonce":4294176576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":4294176576},{"type":"v1","addr":"172.21.15.6:6815","nonce":4294176576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":4294176576},{"type":"v1","addr":"172.21.15.6:6813","nonce":4294176576}]},"public_addr":"172.21.15.6:6809/4294176576","cluster_addr":"172.21.15.6:6811/4294176576","heartbeat_back_addr":"172.21.15.6:6815/4294176576","heartbeat_front_addr":"172.21.15.6:6813/4294176576","state":["exists","up"]},{"osd":1,"uuid":"1d1e25d4-5837-450d-aae1-a44b3233c9ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1789421019},{"type":"v1","addr":"172.21.15.6:6801","nonce":1789421019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1789421019},{"type":"v1","addr":"172.21.15.6:6803","nonce":1789421019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1789421019},{"type":"v1","addr":"172.21.15.6:6807","nonce":1789421019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1789421019},{"type":"v1","addr":"172.21.15.6:6805","nonce":1789421019}]},"public_addr":"172.21.15.6:6801/1789421019","cluster_addr":"172.21.15.6:6803/1789421019","heartbeat_back_addr":"172.21.15.6:6807/1789421019","heartbeat_front_addr":"172.21.15.6:6805/1789421019","state":["exists","up"]},{"osd":2,"uuid":"9540a0fe-29b1-4de7-b617-a765d3f99fc5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":2756875437},{"type":"v1","addr":"172.21.15.6:6817","nonce":2756875437}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":2756875437},{"type":"v1","addr":"172.21.15.6:6819","nonce":2756875437}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":2756875437},{"type":"v1","addr":"172.21.15.6:6823","nonce":2756875437}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":2756875437},{"type":"v1","addr":"172.21.15.6:6821","nonce":2756875437}]},"public_addr":"172.21.15.6:6817/2756875437","cluster_addr":"172.21.15.6:6819/2756875437","heartbeat_back_addr":"172.21.15.6:6823/2756875437","heartbeat_front_addr":"172.21.15.6:6821/2756875437","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T23:04:22.254942+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T23:04:21.768119+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T23:04:21.962217+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T23:04:30.232 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-09-17T23:04:30.233 INFO:tasks.ceph:Creating RBD pool 2024-09-17T23:04:30.233 DEBUG:teuthology.orchestra.run.smithi006:> sudo ceph --cluster cluster1 osd pool create rbd 8 2024-09-17T23:04:30.691 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'rbd' created 2024-09-17T23:04:30.707 DEBUG:teuthology.orchestra.run.smithi006:> rbd --cluster cluster1 pool init rbd 2024-09-17T23:04:33.721 INFO:tasks.ceph:Starting mds daemons in cluster cluster1... 2024-09-17T23:04:33.721 INFO:tasks.ceph:Waiting until cluster1 daemons up and pgs clean... 2024-09-17T23:04:33.722 INFO:tasks.ceph.ceph_manager.cluster1:waiting for mgr available 2024-09-17T23:04:33.722 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 mgr dump --format=json 2024-09-17T23:04:33.722 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-17T23:04:34.070 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:04:34.082 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":5,"flags":0,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":3471003646},{"type":"v1","addr":"172.21.15.6:6825","nonce":3471003646}]},"active_addr":"172.21.15.6:6825/3471003646","active_change":"2024-09-17T23:04:19.507272+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.6:0","nonce":3727726677}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.6:0","nonce":1357374119}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.6:0","nonce":431923625}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.6:0","nonce":839844975}]}]} 2024-09-17T23:04:34.083 INFO:tasks.ceph.ceph_manager.cluster1:mgr available! 2024-09-17T23:04:34.084 INFO:tasks.ceph.ceph_manager.cluster1:waiting for all up 2024-09-17T23:04:34.084 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-09-17T23:04:34.374 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:04:34.375 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":16,"fsid":"545dc37d-b29c-4f04-aad4-46458bcf9f65","created":"2024-09-17T23:04:15.024128+0000","modified":"2024-09-17T23:04:33.705333+0000","last_up_change":"2024-09-17T23:04:24.674100+0000","last_in_change":"2024-09-17T23:04:15.924722+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T23:04:25.756881+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-17T23:04:30.549319+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"530ec6e3-0be9-4615-9d5a-2fa97fce2539","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":4294176576},{"type":"v1","addr":"172.21.15.6:6809","nonce":4294176576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":4294176576},{"type":"v1","addr":"172.21.15.6:6811","nonce":4294176576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":4294176576},{"type":"v1","addr":"172.21.15.6:6815","nonce":4294176576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":4294176576},{"type":"v1","addr":"172.21.15.6:6813","nonce":4294176576}]},"public_addr":"172.21.15.6:6809/4294176576","cluster_addr":"172.21.15.6:6811/4294176576","heartbeat_back_addr":"172.21.15.6:6815/4294176576","heartbeat_front_addr":"172.21.15.6:6813/4294176576","state":["exists","up"]},{"osd":1,"uuid":"1d1e25d4-5837-450d-aae1-a44b3233c9ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1789421019},{"type":"v1","addr":"172.21.15.6:6801","nonce":1789421019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1789421019},{"type":"v1","addr":"172.21.15.6:6803","nonce":1789421019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1789421019},{"type":"v1","addr":"172.21.15.6:6807","nonce":1789421019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1789421019},{"type":"v1","addr":"172.21.15.6:6805","nonce":1789421019}]},"public_addr":"172.21.15.6:6801/1789421019","cluster_addr":"172.21.15.6:6803/1789421019","heartbeat_back_addr":"172.21.15.6:6807/1789421019","heartbeat_front_addr":"172.21.15.6:6805/1789421019","state":["exists","up"]},{"osd":2,"uuid":"9540a0fe-29b1-4de7-b617-a765d3f99fc5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":2756875437},{"type":"v1","addr":"172.21.15.6:6817","nonce":2756875437}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":2756875437},{"type":"v1","addr":"172.21.15.6:6819","nonce":2756875437}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":2756875437},{"type":"v1","addr":"172.21.15.6:6823","nonce":2756875437}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":2756875437},{"type":"v1","addr":"172.21.15.6:6821","nonce":2756875437}]},"public_addr":"172.21.15.6:6817/2756875437","cluster_addr":"172.21.15.6:6819/2756875437","heartbeat_back_addr":"172.21.15.6:6823/2756875437","heartbeat_front_addr":"172.21.15.6:6821/2756875437","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T23:04:22.254942+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T23:04:21.768119+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T23:04:21.962217+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T23:04:34.387 INFO:tasks.ceph.ceph_manager.cluster1:all up! 2024-09-17T23:04:34.387 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-09-17T23:04:34.678 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:04:34.678 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":16,"fsid":"545dc37d-b29c-4f04-aad4-46458bcf9f65","created":"2024-09-17T23:04:15.024128+0000","modified":"2024-09-17T23:04:33.705333+0000","last_up_change":"2024-09-17T23:04:24.674100+0000","last_in_change":"2024-09-17T23:04:15.924722+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T23:04:25.756881+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-17T23:04:30.549319+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"530ec6e3-0be9-4615-9d5a-2fa97fce2539","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":4294176576},{"type":"v1","addr":"172.21.15.6:6809","nonce":4294176576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":4294176576},{"type":"v1","addr":"172.21.15.6:6811","nonce":4294176576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":4294176576},{"type":"v1","addr":"172.21.15.6:6815","nonce":4294176576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":4294176576},{"type":"v1","addr":"172.21.15.6:6813","nonce":4294176576}]},"public_addr":"172.21.15.6:6809/4294176576","cluster_addr":"172.21.15.6:6811/4294176576","heartbeat_back_addr":"172.21.15.6:6815/4294176576","heartbeat_front_addr":"172.21.15.6:6813/4294176576","state":["exists","up"]},{"osd":1,"uuid":"1d1e25d4-5837-450d-aae1-a44b3233c9ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1789421019},{"type":"v1","addr":"172.21.15.6:6801","nonce":1789421019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1789421019},{"type":"v1","addr":"172.21.15.6:6803","nonce":1789421019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1789421019},{"type":"v1","addr":"172.21.15.6:6807","nonce":1789421019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1789421019},{"type":"v1","addr":"172.21.15.6:6805","nonce":1789421019}]},"public_addr":"172.21.15.6:6801/1789421019","cluster_addr":"172.21.15.6:6803/1789421019","heartbeat_back_addr":"172.21.15.6:6807/1789421019","heartbeat_front_addr":"172.21.15.6:6805/1789421019","state":["exists","up"]},{"osd":2,"uuid":"9540a0fe-29b1-4de7-b617-a765d3f99fc5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":2756875437},{"type":"v1","addr":"172.21.15.6:6817","nonce":2756875437}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":2756875437},{"type":"v1","addr":"172.21.15.6:6819","nonce":2756875437}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":2756875437},{"type":"v1","addr":"172.21.15.6:6823","nonce":2756875437}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":2756875437},{"type":"v1","addr":"172.21.15.6:6821","nonce":2756875437}]},"public_addr":"172.21.15.6:6817/2756875437","cluster_addr":"172.21.15.6:6819/2756875437","heartbeat_back_addr":"172.21.15.6:6823/2756875437","heartbeat_front_addr":"172.21.15.6:6821/2756875437","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T23:04:22.254942+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T23:04:21.768119+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T23:04:21.962217+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T23:04:34.691 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.0 flush_pg_stats 2024-09-17T23:04:34.691 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.1 flush_pg_stats 2024-09-17T23:04:34.692 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.2 flush_pg_stats 2024-09-17T23:04:34.838 INFO:teuthology.orchestra.run.smithi006.stdout:34359738373 2024-09-17T23:04:34.838 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.2 2024-09-17T23:04:34.864 INFO:teuthology.orchestra.run.smithi006.stdout:34359738373 2024-09-17T23:04:34.865 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2024-09-17T23:04:34.867 INFO:teuthology.orchestra.run.smithi006.stdout:34359738374 2024-09-17T23:04:34.867 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.1 2024-09-17T23:04:35.136 INFO:teuthology.orchestra.run.smithi006.stdout:34359738372 2024-09-17T23:04:35.149 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738373 got 34359738372 for osd.2 2024-09-17T23:04:35.203 INFO:teuthology.orchestra.run.smithi006.stdout:34359738373 2024-09-17T23:04:35.207 INFO:teuthology.orchestra.run.smithi006.stdout:34359738372 2024-09-17T23:04:35.217 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738374 got 34359738373 for osd.1 2024-09-17T23:04:35.221 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738373 got 34359738372 for osd.0 2024-09-17T23:04:36.150 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.2 2024-09-17T23:04:36.219 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.1 2024-09-17T23:04:36.221 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2024-09-17T23:04:36.466 INFO:teuthology.orchestra.run.smithi006.stdout:34359738373 2024-09-17T23:04:36.479 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738373 got 34359738373 for osd.2 2024-09-17T23:04:36.479 DEBUG:teuthology.parallel:result is None 2024-09-17T23:04:36.522 INFO:teuthology.orchestra.run.smithi006.stdout:34359738374 2024-09-17T23:04:36.534 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738374 got 34359738374 for osd.1 2024-09-17T23:04:36.535 DEBUG:teuthology.parallel:result is None 2024-09-17T23:04:36.563 INFO:teuthology.orchestra.run.smithi006.stdout:34359738373 2024-09-17T23:04:36.575 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738373 got 34359738373 for osd.0 2024-09-17T23:04:36.576 DEBUG:teuthology.parallel:result is None 2024-09-17T23:04:36.576 INFO:tasks.ceph.ceph_manager.cluster1:waiting for clean 2024-09-17T23:04:36.576 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-17T23:04:36.905 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:04:36.905 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-09-17T23:04:36.917 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2024-09-17T23:04:35.513451+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":81612,"kb_used_data":520,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283033908,"statfs":{"total":289910292480,"available":289826721792,"internally_reserved":0,"allocated":532480,"data_stored":1299600,"data_compressed":5796,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4785,"internal_metadata":82963791},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.822773"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-17T23:04:33.715156+0000","last_change":"2024-09-17T23:04:33.715156+0000","last_active":"2024-09-17T23:04:33.715156+0000","last_peered":"2024-09-17T23:04:33.715156+0000","last_clean":"2024-09-17T23:04:33.715156+0000","last_became_active":"2024-09-17T23:04:31.700751+0000","last_became_peered":"2024-09-17T23:04:31.700751+0000","last_unstale":"2024-09-17T23:04:33.715156+0000","last_undegraded":"2024-09-17T23:04:33.715156+0000","last_fullsized":"2024-09-17T23:04:33.715156+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:04:30.688961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:04:30.688961+0000","last_clean_scrub_stamp":"2024-09-17T23:04:30.688961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T00:01:28.967464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00096591400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-17T23:04:33.714782+0000","last_change":"2024-09-17T23:04:33.714782+0000","last_active":"2024-09-17T23:04:33.714782+0000","last_peered":"2024-09-17T23:04:33.714782+0000","last_clean":"2024-09-17T23:04:33.714782+0000","last_became_active":"2024-09-17T23:04:31.701763+0000","last_became_peered":"2024-09-17T23:04:31.701763+0000","last_unstale":"2024-09-17T23:04:33.714782+0000","last_undegraded":"2024-09-17T23:04:33.714782+0000","last_fullsized":"2024-09-17T23:04:33.714782+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:04:30.688961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:04:30.688961+0000","last_clean_scrub_stamp":"2024-09-17T23:04:30.688961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T05:25:40.855629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065844999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-17T23:04:33.715201+0000","last_change":"2024-09-17T23:04:33.715201+0000","last_active":"2024-09-17T23:04:33.715201+0000","last_peered":"2024-09-17T23:04:33.715201+0000","last_clean":"2024-09-17T23:04:33.715201+0000","last_became_active":"2024-09-17T23:04:31.703638+0000","last_became_peered":"2024-09-17T23:04:31.703638+0000","last_unstale":"2024-09-17T23:04:33.715201+0000","last_undegraded":"2024-09-17T23:04:33.715201+0000","last_fullsized":"2024-09-17T23:04:33.715201+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:04:30.688961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:04:30.688961+0000","last_clean_scrub_stamp":"2024-09-17T23:04:30.688961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T06:57:29.183690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097086099999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-17T23:04:33.714901+0000","last_change":"2024-09-17T23:04:33.714901+0000","last_active":"2024-09-17T23:04:33.714901+0000","last_peered":"2024-09-17T23:04:33.714901+0000","last_clean":"2024-09-17T23:04:33.714901+0000","last_became_active":"2024-09-17T23:04:31.702321+0000","last_became_peered":"2024-09-17T23:04:31.702321+0000","last_unstale":"2024-09-17T23:04:33.714901+0000","last_undegraded":"2024-09-17T23:04:33.714901+0000","last_fullsized":"2024-09-17T23:04:33.714901+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:04:30.688961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:04:30.688961+0000","last_clean_scrub_stamp":"2024-09-17T23:04:30.688961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T09:51:38.352425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00072955299999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"16'2","reported_seq":21,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-17T23:04:33.713584+0000","last_change":"2024-09-17T23:04:33.710732+0000","last_active":"2024-09-17T23:04:33.713584+0000","last_peered":"2024-09-17T23:04:33.713584+0000","last_clean":"2024-09-17T23:04:33.713584+0000","last_became_active":"2024-09-17T23:04:31.701189+0000","last_became_peered":"2024-09-17T23:04:31.701189+0000","last_unstale":"2024-09-17T23:04:33.713584+0000","last_undegraded":"2024-09-17T23:04:33.713584+0000","last_fullsized":"2024-09-17T23:04:33.713584+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:04:30.688961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:04:30.688961+0000","last_clean_scrub_stamp":"2024-09-17T23:04:30.688961+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T02:38:23.106589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010931420000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-17T23:04:33.939593+0000","last_change":"2024-09-17T23:04:33.939593+0000","last_active":"2024-09-17T23:04:33.939593+0000","last_peered":"2024-09-17T23:04:33.939593+0000","last_clean":"2024-09-17T23:04:33.939593+0000","last_became_active":"2024-09-17T23:04:31.701197+0000","last_became_peered":"2024-09-17T23:04:31.701197+0000","last_unstale":"2024-09-17T23:04:33.939593+0000","last_undegraded":"2024-09-17T23:04:33.939593+0000","last_fullsized":"2024-09-17T23:04:33.939593+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:04:30.688961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:04:30.688961+0000","last_clean_scrub_stamp":"2024-09-17T23:04:30.688961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T08:19:18.099975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032506599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-17T23:04:33.939656+0000","last_change":"2024-09-17T23:04:33.939656+0000","last_active":"2024-09-17T23:04:33.939656+0000","last_peered":"2024-09-17T23:04:33.939656+0000","last_clean":"2024-09-17T23:04:33.939656+0000","last_became_active":"2024-09-17T23:04:31.702476+0000","last_became_peered":"2024-09-17T23:04:31.702476+0000","last_unstale":"2024-09-17T23:04:33.939656+0000","last_undegraded":"2024-09-17T23:04:33.939656+0000","last_fullsized":"2024-09-17T23:04:33.939656+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:04:30.688961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:04:30.688961+0000","last_clean_scrub_stamp":"2024-09-17T23:04:30.688961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T06:57:19.469661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038066100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"14'1","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-17T23:04:33.714583+0000","last_change":"2024-09-17T23:04:33.714583+0000","last_active":"2024-09-17T23:04:33.714583+0000","last_peered":"2024-09-17T23:04:33.714583+0000","last_clean":"2024-09-17T23:04:33.714583+0000","last_became_active":"2024-09-17T23:04:31.700794+0000","last_became_peered":"2024-09-17T23:04:31.700794+0000","last_unstale":"2024-09-17T23:04:33.714583+0000","last_undegraded":"2024-09-17T23:04:33.714583+0000","last_fullsized":"2024-09-17T23:04:33.714583+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:04:30.688961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:04:30.688961+0000","last_clean_scrub_stamp":"2024-09-17T23:04:30.688961+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T09:53:38.917955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051287900000000005,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"11'97","reported_seq":146,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-17T23:04:33.713978+0000","last_change":"2024-09-17T23:04:27.966616+0000","last_active":"2024-09-17T23:04:33.713978+0000","last_peered":"2024-09-17T23:04:33.713978+0000","last_clean":"2024-09-17T23:04:33.713978+0000","last_became_active":"2024-09-17T23:04:27.965566+0000","last_became_peered":"2024-09-17T23:04:27.965566+0000","last_unstale":"2024-09-17T23:04:33.713978+0000","last_undegraded":"2024-09-17T23:04:33.713978+0000","last_fullsized":"2024-09-17T23:04:33.713978+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:04:26.677035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:04:26.677035+0000","last_clean_scrub_stamp":"2024-09-17T23:04:26.677035+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T10:58:45.080759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":5796,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27156,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344684,"statfs":{"total":96636764160,"available":96608956416,"internally_reserved":0,"allocated":131072,"data_stored":37234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738374,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":200704,"data_stored":631183,"data_compressed":2898,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738373,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":200704,"data_stored":631183,"data_compressed":2898,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2898,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2898,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T23:04:36.918 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-17T23:04:37.203 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:04:37.203 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-09-17T23:04:37.215 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2024-09-17T23:04:35.513451+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":81612,"kb_used_data":520,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283033908,"statfs":{"total":289910292480,"available":289826721792,"internally_reserved":0,"allocated":532480,"data_stored":1299600,"data_compressed":5796,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4785,"internal_metadata":82963791},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.822773"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-17T23:04:33.715156+0000","last_change":"2024-09-17T23:04:33.715156+0000","last_active":"2024-09-17T23:04:33.715156+0000","last_peered":"2024-09-17T23:04:33.715156+0000","last_clean":"2024-09-17T23:04:33.715156+0000","last_became_active":"2024-09-17T23:04:31.700751+0000","last_became_peered":"2024-09-17T23:04:31.700751+0000","last_unstale":"2024-09-17T23:04:33.715156+0000","last_undegraded":"2024-09-17T23:04:33.715156+0000","last_fullsized":"2024-09-17T23:04:33.715156+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:04:30.688961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:04:30.688961+0000","last_clean_scrub_stamp":"2024-09-17T23:04:30.688961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T00:01:28.967464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00096591400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-17T23:04:33.714782+0000","last_change":"2024-09-17T23:04:33.714782+0000","last_active":"2024-09-17T23:04:33.714782+0000","last_peered":"2024-09-17T23:04:33.714782+0000","last_clean":"2024-09-17T23:04:33.714782+0000","last_became_active":"2024-09-17T23:04:31.701763+0000","last_became_peered":"2024-09-17T23:04:31.701763+0000","last_unstale":"2024-09-17T23:04:33.714782+0000","last_undegraded":"2024-09-17T23:04:33.714782+0000","last_fullsized":"2024-09-17T23:04:33.714782+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:04:30.688961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:04:30.688961+0000","last_clean_scrub_stamp":"2024-09-17T23:04:30.688961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T05:25:40.855629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065844999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-17T23:04:33.715201+0000","last_change":"2024-09-17T23:04:33.715201+0000","last_active":"2024-09-17T23:04:33.715201+0000","last_peered":"2024-09-17T23:04:33.715201+0000","last_clean":"2024-09-17T23:04:33.715201+0000","last_became_active":"2024-09-17T23:04:31.703638+0000","last_became_peered":"2024-09-17T23:04:31.703638+0000","last_unstale":"2024-09-17T23:04:33.715201+0000","last_undegraded":"2024-09-17T23:04:33.715201+0000","last_fullsized":"2024-09-17T23:04:33.715201+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:04:30.688961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:04:30.688961+0000","last_clean_scrub_stamp":"2024-09-17T23:04:30.688961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T06:57:29.183690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097086099999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-17T23:04:33.714901+0000","last_change":"2024-09-17T23:04:33.714901+0000","last_active":"2024-09-17T23:04:33.714901+0000","last_peered":"2024-09-17T23:04:33.714901+0000","last_clean":"2024-09-17T23:04:33.714901+0000","last_became_active":"2024-09-17T23:04:31.702321+0000","last_became_peered":"2024-09-17T23:04:31.702321+0000","last_unstale":"2024-09-17T23:04:33.714901+0000","last_undegraded":"2024-09-17T23:04:33.714901+0000","last_fullsized":"2024-09-17T23:04:33.714901+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:04:30.688961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:04:30.688961+0000","last_clean_scrub_stamp":"2024-09-17T23:04:30.688961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T09:51:38.352425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00072955299999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"16'2","reported_seq":21,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-17T23:04:33.713584+0000","last_change":"2024-09-17T23:04:33.710732+0000","last_active":"2024-09-17T23:04:33.713584+0000","last_peered":"2024-09-17T23:04:33.713584+0000","last_clean":"2024-09-17T23:04:33.713584+0000","last_became_active":"2024-09-17T23:04:31.701189+0000","last_became_peered":"2024-09-17T23:04:31.701189+0000","last_unstale":"2024-09-17T23:04:33.713584+0000","last_undegraded":"2024-09-17T23:04:33.713584+0000","last_fullsized":"2024-09-17T23:04:33.713584+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:04:30.688961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:04:30.688961+0000","last_clean_scrub_stamp":"2024-09-17T23:04:30.688961+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T02:38:23.106589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010931420000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-17T23:04:33.939593+0000","last_change":"2024-09-17T23:04:33.939593+0000","last_active":"2024-09-17T23:04:33.939593+0000","last_peered":"2024-09-17T23:04:33.939593+0000","last_clean":"2024-09-17T23:04:33.939593+0000","last_became_active":"2024-09-17T23:04:31.701197+0000","last_became_peered":"2024-09-17T23:04:31.701197+0000","last_unstale":"2024-09-17T23:04:33.939593+0000","last_undegraded":"2024-09-17T23:04:33.939593+0000","last_fullsized":"2024-09-17T23:04:33.939593+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:04:30.688961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:04:30.688961+0000","last_clean_scrub_stamp":"2024-09-17T23:04:30.688961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T08:19:18.099975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032506599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-17T23:04:33.939656+0000","last_change":"2024-09-17T23:04:33.939656+0000","last_active":"2024-09-17T23:04:33.939656+0000","last_peered":"2024-09-17T23:04:33.939656+0000","last_clean":"2024-09-17T23:04:33.939656+0000","last_became_active":"2024-09-17T23:04:31.702476+0000","last_became_peered":"2024-09-17T23:04:31.702476+0000","last_unstale":"2024-09-17T23:04:33.939656+0000","last_undegraded":"2024-09-17T23:04:33.939656+0000","last_fullsized":"2024-09-17T23:04:33.939656+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:04:30.688961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:04:30.688961+0000","last_clean_scrub_stamp":"2024-09-17T23:04:30.688961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T06:57:19.469661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038066100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"14'1","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-17T23:04:33.714583+0000","last_change":"2024-09-17T23:04:33.714583+0000","last_active":"2024-09-17T23:04:33.714583+0000","last_peered":"2024-09-17T23:04:33.714583+0000","last_clean":"2024-09-17T23:04:33.714583+0000","last_became_active":"2024-09-17T23:04:31.700794+0000","last_became_peered":"2024-09-17T23:04:31.700794+0000","last_unstale":"2024-09-17T23:04:33.714583+0000","last_undegraded":"2024-09-17T23:04:33.714583+0000","last_fullsized":"2024-09-17T23:04:33.714583+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:04:30.688961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:04:30.688961+0000","last_clean_scrub_stamp":"2024-09-17T23:04:30.688961+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T09:53:38.917955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051287900000000005,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"11'97","reported_seq":146,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-17T23:04:33.713978+0000","last_change":"2024-09-17T23:04:27.966616+0000","last_active":"2024-09-17T23:04:33.713978+0000","last_peered":"2024-09-17T23:04:33.713978+0000","last_clean":"2024-09-17T23:04:33.713978+0000","last_became_active":"2024-09-17T23:04:27.965566+0000","last_became_peered":"2024-09-17T23:04:27.965566+0000","last_unstale":"2024-09-17T23:04:33.713978+0000","last_undegraded":"2024-09-17T23:04:33.713978+0000","last_fullsized":"2024-09-17T23:04:33.713978+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:04:26.677035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:04:26.677035+0000","last_clean_scrub_stamp":"2024-09-17T23:04:26.677035+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T10:58:45.080759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":5796,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27156,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344684,"statfs":{"total":96636764160,"available":96608956416,"internally_reserved":0,"allocated":131072,"data_stored":37234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738374,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":200704,"data_stored":631183,"data_compressed":2898,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738373,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":200704,"data_stored":631183,"data_compressed":2898,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2898,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2898,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T23:04:37.215 INFO:tasks.ceph.ceph_manager.cluster1:clean! 2024-09-17T23:04:37.216 INFO:tasks.ceph:Waiting until ceph cluster cluster1 is healthy... 2024-09-17T23:04:37.216 INFO:tasks.ceph.ceph_manager.cluster1:wait_until_healthy 2024-09-17T23:04:37.216 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 health --format=json 2024-09-17T23:04:37.545 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:04:37.546 INFO:teuthology.orchestra.run.smithi006.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-17T23:04:37.558 INFO:tasks.ceph.ceph_manager.cluster1:wait_until_healthy done 2024-09-17T23:04:37.559 INFO:teuthology.run_tasks:Running task ceph... 2024-09-17T23:04:37.566 INFO:tasks.ceph:Creating ceph cluster cluster2... 2024-09-17T23:04:37.566 INFO:tasks.ceph:config {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'cluster2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-17T23:04:37.567 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-09-17_21:16:02-rbd-squid-distro-default-smithi/7909572', 'branch': 'squid', 'description': 'rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-zstd policy/none rbd-mirror/four-per-cluster supported-random-distro$/{centos_latest} workloads/rbd-mirror-snapshot-stress-workunit-fast-diff}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7909572', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'meta': [{'desc': 'run two ceph clusters and install rbd-mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': '2 ceph clusters with 1 mon and 3 osds each'}, {'desc': 'run four rbd-mirror daemons per cluster'}, {'desc': 'run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon'}], 'name': 'teuthology-2024-09-17_21:16:02-rbd-squid-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'squid'}, 'ceph': {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'c2ddf9721ef254645201cdbabd9c8f49c30a586f'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'c2ddf9721ef254645201cdbabd9c8f49c30a586f'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'squid', 'sha1': 'b0d356b98e08f6bc59ef15652f0d0e4298fc83c7'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 100, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'], ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6']], 'seed': 1983, 'sha1': 'c2ddf9721ef254645201cdbabd9c8f49c30a586f', 'sleep_before_teardown': 0, 'subset': '79/128', 'suite': 'rbd', 'suite_branch': 'squid', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_b0d356b98e08f6bc59ef15652f0d0e4298fc83c7/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': 'b0d356b98e08f6bc59ef15652f0d0e4298fc83c7', 'targets': {'smithi006.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBI25UHMjorqH5xXKrb+xjNLrKzHw3AquyMACLWNnQVzXSAKmfZF29gIWsRpW0CsJC8C7QZbDaT9NmJu+Vqwsk7o=', 'smithi038.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE1eTtc2bdOQzb/4YSSSQQ8mKGiFq5T8t3u2PuUJ7oED7RK6GtrnDZsTnuisjLHXzLmeaYHBzMxFHTTfoD/Zn6Y='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': 'c2ddf9721ef254645201cdbabd9c8f49c30a586f'}}, {'ceph': {'cluster': 'cluster1', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'c2ddf9721ef254645201cdbabd9c8f49c30a586f'}}, {'ceph': {'cluster': 'cluster2', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'c2ddf9721ef254645201cdbabd9c8f49c30a586f'}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.3', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.3', 'thrash': True}}, {'rbd-mirror-thrash': {'cluster': 'cluster1'}}, {'rbd-mirror-thrash': {'cluster': 'cluster2'}}, {'workunit': {'clients': {'cluster1.client.mirror': ['rbd/rbd_mirror_stress.sh']}, 'env': {'CEPH_ARGS': '', 'MIRROR_POOL_MODE': 'image', 'RBD_IMAGE_FEATURES': 'layering,exclusive-lock,object-map,fast-diff', 'RBD_MIRROR_INSTANCES': '4', 'RBD_MIRROR_MODE': 'snapshot', 'RBD_MIRROR_USE_EXISTING_CLUSTER': '1', 'RBD_MIRROR_USE_RBD_MIRROR': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-17_21:16:02', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-17T23:04:37.567 DEBUG:teuthology.orchestra.run.smithi006:> install -d -m0755 -- /home/ubuntu/cephtest/cluster2.data 2024-09-17T23:04:37.570 DEBUG:teuthology.orchestra.run.smithi038:> install -d -m0755 -- /home/ubuntu/cephtest/cluster2.data 2024-09-17T23:04:37.589 DEBUG:teuthology.orchestra.run.smithi006:> sudo install -d -m0777 -- /var/run/ceph 2024-09-17T23:04:37.625 DEBUG:teuthology.orchestra.run.smithi038:> sudo install -d -m0777 -- /var/run/ceph 2024-09-17T23:04:37.660 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-09-17T23:04:37.660 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/scratch_devs of=/dev/stdout 2024-09-17T23:04:37.719 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-17T23:04:37.719 DEBUG:teuthology.orchestra.run.smithi038:> stat /dev/vg_nvme/lv_1 2024-09-17T23:04:37.777 INFO:teuthology.orchestra.run.smithi038.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-17T23:04:37.777 INFO:teuthology.orchestra.run.smithi038.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T23:04:37.777 INFO:teuthology.orchestra.run.smithi038.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-09-17T23:04:37.777 INFO:teuthology.orchestra.run.smithi038.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T23:04:37.777 INFO:teuthology.orchestra.run.smithi038.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T23:04:37.777 INFO:teuthology.orchestra.run.smithi038.stdout:Access: 2024-09-17 23:03:51.089832578 +0000 2024-09-17T23:04:37.777 INFO:teuthology.orchestra.run.smithi038.stdout:Modify: 2024-09-17 23:01:10.884575211 +0000 2024-09-17T23:04:37.777 INFO:teuthology.orchestra.run.smithi038.stdout:Change: 2024-09-17 23:01:10.884575211 +0000 2024-09-17T23:04:37.777 INFO:teuthology.orchestra.run.smithi038.stdout: Birth: 2024-09-17 23:01:10.884575211 +0000 2024-09-17T23:04:37.778 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-17T23:04:37.845 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records in 2024-09-17T23:04:37.845 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records out 2024-09-17T23:04:37.845 INFO:teuthology.orchestra.run.smithi038.stderr:512 bytes copied, 0.000381268 s, 1.3 MB/s 2024-09-17T23:04:37.847 DEBUG:teuthology.orchestra.run.smithi038:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-17T23:04:37.905 DEBUG:teuthology.orchestra.run.smithi038:> stat /dev/vg_nvme/lv_2 2024-09-17T23:04:37.962 INFO:teuthology.orchestra.run.smithi038.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-17T23:04:37.963 INFO:teuthology.orchestra.run.smithi038.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T23:04:37.963 INFO:teuthology.orchestra.run.smithi038.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-09-17T23:04:37.963 INFO:teuthology.orchestra.run.smithi038.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T23:04:37.963 INFO:teuthology.orchestra.run.smithi038.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T23:04:37.963 INFO:teuthology.orchestra.run.smithi038.stdout:Access: 2024-09-17 23:03:51.090832560 +0000 2024-09-17T23:04:37.963 INFO:teuthology.orchestra.run.smithi038.stdout:Modify: 2024-09-17 23:01:11.226569117 +0000 2024-09-17T23:04:37.963 INFO:teuthology.orchestra.run.smithi038.stdout:Change: 2024-09-17 23:01:11.226569117 +0000 2024-09-17T23:04:37.963 INFO:teuthology.orchestra.run.smithi038.stdout: Birth: 2024-09-17 23:01:11.226569117 +0000 2024-09-17T23:04:37.963 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-17T23:04:38.030 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records in 2024-09-17T23:04:38.030 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records out 2024-09-17T23:04:38.030 INFO:teuthology.orchestra.run.smithi038.stderr:512 bytes copied, 0.000329923 s, 1.6 MB/s 2024-09-17T23:04:38.032 DEBUG:teuthology.orchestra.run.smithi038:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-17T23:04:38.089 DEBUG:teuthology.orchestra.run.smithi038:> stat /dev/vg_nvme/lv_3 2024-09-17T23:04:38.145 INFO:teuthology.orchestra.run.smithi038.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-17T23:04:38.145 INFO:teuthology.orchestra.run.smithi038.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T23:04:38.145 INFO:teuthology.orchestra.run.smithi038.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-09-17T23:04:38.146 INFO:teuthology.orchestra.run.smithi038.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T23:04:38.146 INFO:teuthology.orchestra.run.smithi038.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T23:04:38.146 INFO:teuthology.orchestra.run.smithi038.stdout:Access: 2024-09-17 23:03:51.090832560 +0000 2024-09-17T23:04:38.146 INFO:teuthology.orchestra.run.smithi038.stdout:Modify: 2024-09-17 23:01:11.558563202 +0000 2024-09-17T23:04:38.146 INFO:teuthology.orchestra.run.smithi038.stdout:Change: 2024-09-17 23:01:11.558563202 +0000 2024-09-17T23:04:38.146 INFO:teuthology.orchestra.run.smithi038.stdout: Birth: 2024-09-17 23:01:11.558563202 +0000 2024-09-17T23:04:38.146 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-17T23:04:38.210 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records in 2024-09-17T23:04:38.210 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records out 2024-09-17T23:04:38.210 INFO:teuthology.orchestra.run.smithi038.stderr:512 bytes copied, 0.000315529 s, 1.6 MB/s 2024-09-17T23:04:38.211 DEBUG:teuthology.orchestra.run.smithi038:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-17T23:04:38.268 DEBUG:teuthology.orchestra.run.smithi038:> stat /dev/vg_nvme/lv_4 2024-09-17T23:04:38.324 INFO:teuthology.orchestra.run.smithi038.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-17T23:04:38.325 INFO:teuthology.orchestra.run.smithi038.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T23:04:38.325 INFO:teuthology.orchestra.run.smithi038.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-09-17T23:04:38.325 INFO:teuthology.orchestra.run.smithi038.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T23:04:38.325 INFO:teuthology.orchestra.run.smithi038.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T23:04:38.325 INFO:teuthology.orchestra.run.smithi038.stdout:Access: 2024-09-17 23:03:51.091832542 +0000 2024-09-17T23:04:38.325 INFO:teuthology.orchestra.run.smithi038.stdout:Modify: 2024-09-17 23:01:11.882557429 +0000 2024-09-17T23:04:38.325 INFO:teuthology.orchestra.run.smithi038.stdout:Change: 2024-09-17 23:01:11.882557429 +0000 2024-09-17T23:04:38.325 INFO:teuthology.orchestra.run.smithi038.stdout: Birth: 2024-09-17 23:01:11.882557429 +0000 2024-09-17T23:04:38.325 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-17T23:04:38.388 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records in 2024-09-17T23:04:38.389 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records out 2024-09-17T23:04:38.389 INFO:teuthology.orchestra.run.smithi038.stderr:512 bytes copied, 0.000320322 s, 1.6 MB/s 2024-09-17T23:04:38.390 DEBUG:teuthology.orchestra.run.smithi038:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-17T23:04:38.448 INFO:tasks.ceph:osd dev map: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-17T23:04:38.449 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi038.front.sepia.ceph.com'): {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-17T23:04:38.449 INFO:tasks.ceph:Generating config... 2024-09-17T23:04:38.450 INFO:tasks.ceph:[client] debug rbd = 20 2024-09-17T23:04:38.450 INFO:tasks.ceph:[client] debug rbd_mirror = 15 2024-09-17T23:04:38.450 INFO:tasks.ceph:[client] log to stderr = False 2024-09-17T23:04:38.450 INFO:tasks.ceph:[client] rbd default features = 125 2024-09-17T23:04:38.450 INFO:tasks.ceph:[client] rbd mirror image policy type = none 2024-09-17T23:04:38.450 INFO:tasks.ceph:[client.mirror.0] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-17T23:04:38.450 INFO:tasks.ceph:[client.mirror.0] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-17T23:04:38.450 INFO:tasks.ceph:[client.mirror.1] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-17T23:04:38.450 INFO:tasks.ceph:[client.mirror.1] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-17T23:04:38.450 INFO:tasks.ceph:[client.mirror.2] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-17T23:04:38.451 INFO:tasks.ceph:[client.mirror.2] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-17T23:04:38.451 INFO:tasks.ceph:[client.mirror.3] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-17T23:04:38.451 INFO:tasks.ceph:[client.mirror.3] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-17T23:04:38.451 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-09-17T23:04:38.451 INFO:tasks.ceph:[global] mon warn on pool no app = False 2024-09-17T23:04:38.451 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-09-17T23:04:38.451 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-17T23:04:38.451 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-17T23:04:38.451 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-17T23:04:38.451 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-17T23:04:38.451 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-17T23:04:38.451 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-17T23:04:38.451 INFO:tasks.ceph:[osd] bluestore compression algorithm = zstd 2024-09-17T23:04:38.451 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-09-17T23:04:38.451 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-17T23:04:38.452 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-09-17T23:04:38.452 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-09-17T23:04:38.452 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-17T23:04:38.452 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-17T23:04:38.452 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-09-17T23:04:38.452 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-17T23:04:38.452 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-17T23:04:38.452 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-17T23:04:38.452 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-17T23:04:38.452 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-17T23:04:38.452 INFO:tasks.ceph:Setting up cluster2.mon.a... 2024-09-17T23:04:38.453 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/cluster2.keyring 2024-09-17T23:04:38.530 INFO:teuthology.orchestra.run.smithi038.stdout:creating /etc/ceph/cluster2.keyring 2024-09-17T23:04:38.533 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/cluster2.keyring 2024-09-17T23:04:38.618 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2024-09-17T23:04:38.644 DEBUG:tasks.ceph:Ceph mon addresses: [('cluster2.mon.a', '172.21.15.38')] 2024-09-17T23:04:38.645 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.38', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'mon.a': {}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}} 2024-09-17T23:04:38.645 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-09-17T23:04:38.645 DEBUG:teuthology.orchestra.run.smithi038:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-17T23:04:38.700 DEBUG:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.38 --print /home/ubuntu/cephtest/cluster2.monmap 2024-09-17T23:04:38.781 INFO:teuthology.orchestra.run.smithi038.stdout:monmaptool: monmap file /home/ubuntu/cephtest/cluster2.monmap 2024-09-17T23:04:38.782 INFO:teuthology.orchestra.run.smithi038.stdout:monmaptool: generated fsid 9e14216a-77a4-4aa5-9521-b021dc9cf0cd 2024-09-17T23:04:38.782 INFO:teuthology.orchestra.run.smithi038.stdout:setting min_mon_release = quincy 2024-09-17T23:04:38.782 INFO:teuthology.orchestra.run.smithi038.stdout:epoch 0 2024-09-17T23:04:38.782 INFO:teuthology.orchestra.run.smithi038.stdout:fsid 9e14216a-77a4-4aa5-9521-b021dc9cf0cd 2024-09-17T23:04:38.782 INFO:teuthology.orchestra.run.smithi038.stdout:last_changed 2024-09-17T23:04:38.780286+0000 2024-09-17T23:04:38.782 INFO:teuthology.orchestra.run.smithi038.stdout:created 2024-09-17T23:04:38.780286+0000 2024-09-17T23:04:38.782 INFO:teuthology.orchestra.run.smithi038.stdout:min_mon_release 17 (quincy) 2024-09-17T23:04:38.782 INFO:teuthology.orchestra.run.smithi038.stdout:election_strategy: 1 2024-09-17T23:04:38.782 INFO:teuthology.orchestra.run.smithi038.stdout:0: [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] mon.a 2024-09-17T23:04:38.782 INFO:teuthology.orchestra.run.smithi038.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/cluster2.monmap (1 monitors) 2024-09-17T23:04:38.783 DEBUG:teuthology.orchestra.run.smithi038:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-17T23:04:38.839 INFO:tasks.ceph:Writing /etc/ceph/cluster2.conf for FSID 9e14216a-77a4-4aa5-9521-b021dc9cf0cd... 2024-09-17T23:04:38.840 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster2.conf && sudo chmod 0644 /etc/ceph/cluster2.conf > /dev/null 2024-09-17T23:04:38.842 DEBUG:teuthology.orchestra.run.smithi038:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster2.conf && sudo chmod 0644 /etc/ceph/cluster2.conf > /dev/null 2024-09-17T23:04:38.893 INFO:teuthology.orchestra.run.smithi006.stdout:[global] 2024-09-17T23:04:38.893 INFO:teuthology.orchestra.run.smithi006.stdout: chdir = "" 2024-09-17T23:04:38.893 INFO:teuthology.orchestra.run.smithi006.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-17T23:04:38.893 INFO:teuthology.orchestra.run.smithi006.stdout: auth supported = cephx 2024-09-17T23:04:38.893 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:04:38.893 INFO:teuthology.orchestra.run.smithi006.stdout: filestore xattr use omap = true 2024-09-17T23:04:38.893 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:04:38.894 INFO:teuthology.orchestra.run.smithi006.stdout: mon clock drift allowed = 1.000 2024-09-17T23:04:38.894 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:04:38.894 INFO:teuthology.orchestra.run.smithi006.stdout: osd crush chooseleaf type = 0 2024-09-17T23:04:38.894 INFO:teuthology.orchestra.run.smithi006.stdout: auth debug = true 2024-09-17T23:04:38.894 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:04:38.894 INFO:teuthology.orchestra.run.smithi006.stdout: ms die on old message = true 2024-09-17T23:04:38.894 INFO:teuthology.orchestra.run.smithi006.stdout: ms die on bug = true 2024-09-17T23:04:38.894 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:04:38.894 INFO:teuthology.orchestra.run.smithi006.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-17T23:04:38.894 INFO:teuthology.orchestra.run.smithi006.stdout: mon pg warn max object skew = 0 2024-09-17T23:04:38.894 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:04:38.894 INFO:teuthology.orchestra.run.smithi006.stdout: # disable pg_autoscaler by default for new pools 2024-09-17T23:04:38.894 INFO:teuthology.orchestra.run.smithi006.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-17T23:04:38.894 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:04:38.894 INFO:teuthology.orchestra.run.smithi006.stdout: osd pool default size = 2 2024-09-17T23:04:38.895 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:04:38.895 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd allow primary affinity = true 2024-09-17T23:04:38.895 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd allow pg remap = true 2024-09-17T23:04:38.895 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on legacy crush tunables = false 2024-09-17T23:04:38.895 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on crush straw calc version zero = false 2024-09-17T23:04:38.895 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on no sortbitwise = false 2024-09-17T23:04:38.895 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on osd down out interval zero = false 2024-09-17T23:04:38.895 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on too few osds = false 2024-09-17T23:04:38.895 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-17T23:04:38.895 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-17T23:04:38.895 INFO:teuthology.orchestra.run.smithi006.stdout: mon_allow_pool_size_one = true 2024-09-17T23:04:38.895 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:04:38.896 INFO:teuthology.orchestra.run.smithi006.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-17T23:04:38.896 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:04:38.896 INFO:teuthology.orchestra.run.smithi006.stdout: osd default data pool replay window = 5 2024-09-17T23:04:38.896 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:04:38.896 INFO:teuthology.orchestra.run.smithi006.stdout: mon allow pool delete = true 2024-09-17T23:04:38.896 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:04:38.896 INFO:teuthology.orchestra.run.smithi006.stdout: mon cluster log file level = debug 2024-09-17T23:04:38.896 INFO:teuthology.orchestra.run.smithi006.stdout: debug asserts on shutdown = true 2024-09-17T23:04:38.896 INFO:teuthology.orchestra.run.smithi006.stdout: mon health detail to clog = false 2024-09-17T23:04:38.896 INFO:teuthology.orchestra.run.smithi006.stdout: mon host = 172.21.15.38 2024-09-17T23:04:38.897 INFO:teuthology.orchestra.run.smithi006.stdout: mon client directed command retry = 5 2024-09-17T23:04:38.897 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on pool no app = False 2024-09-17T23:04:38.897 INFO:teuthology.orchestra.run.smithi006.stdout: ms inject socket failures = 5000 2024-09-17T23:04:38.897 INFO:teuthology.orchestra.run.smithi006.stdout: fsid = 9e14216a-77a4-4aa5-9521-b021dc9cf0cd 2024-09-17T23:04:38.897 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:04:38.897 INFO:teuthology.orchestra.run.smithi006.stdout:[osd] 2024-09-17T23:04:38.897 INFO:teuthology.orchestra.run.smithi006.stdout: osd journal size = 100 2024-09-17T23:04:38.897 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:04:38.897 INFO:teuthology.orchestra.run.smithi006.stdout: osd scrub load threshold = 5.0 2024-09-17T23:04:38.897 INFO:teuthology.orchestra.run.smithi006.stdout: osd scrub max interval = 600 2024-09-17T23:04:38.897 INFO:teuthology.orchestra.run.smithi006.stdout: osd mclock profile = high_recovery_ops 2024-09-17T23:04:38.897 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:04:38.897 INFO:teuthology.orchestra.run.smithi006.stdout: osd recover clone overlap = true 2024-09-17T23:04:38.897 INFO:teuthology.orchestra.run.smithi006.stdout: osd recovery max chunk = 1048576 2024-09-17T23:04:38.897 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:04:38.898 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug shutdown = true 2024-09-17T23:04:38.898 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug op order = true 2024-09-17T23:04:38.898 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug verify stray on activate = true 2024-09-17T23:04:38.898 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug trim objects = true 2024-09-17T23:04:38.898 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:04:38.898 INFO:teuthology.orchestra.run.smithi006.stdout: osd open classes on start = true 2024-09-17T23:04:38.898 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug pg log writeout = true 2024-09-17T23:04:38.898 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:04:38.898 INFO:teuthology.orchestra.run.smithi006.stdout: osd deep scrub update digest min age = 30 2024-09-17T23:04:38.898 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:04:38.898 INFO:teuthology.orchestra.run.smithi006.stdout: osd map max advance = 10 2024-09-17T23:04:38.898 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:04:38.898 INFO:teuthology.orchestra.run.smithi006.stdout: journal zero on create = true 2024-09-17T23:04:38.898 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:04:38.898 INFO:teuthology.orchestra.run.smithi006.stdout: filestore ondisk finisher threads = 3 2024-09-17T23:04:38.898 INFO:teuthology.orchestra.run.smithi006.stdout: filestore apply finisher threads = 3 2024-09-17T23:04:38.899 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:04:38.899 INFO:teuthology.orchestra.run.smithi006.stdout: bdev debug aio = true 2024-09-17T23:04:38.899 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug misdirected ops = true 2024-09-17T23:04:38.899 INFO:teuthology.orchestra.run.smithi006.stdout: bluestore block size = 96636764160 2024-09-17T23:04:38.899 INFO:teuthology.orchestra.run.smithi006.stdout: bluestore compression algorithm = zstd 2024-09-17T23:04:38.899 INFO:teuthology.orchestra.run.smithi006.stdout: bluestore compression mode = aggressive 2024-09-17T23:04:38.899 INFO:teuthology.orchestra.run.smithi006.stdout: bluestore fsck on mount = True 2024-09-17T23:04:38.899 INFO:teuthology.orchestra.run.smithi006.stdout: debug bluefs = 1/20 2024-09-17T23:04:38.899 INFO:teuthology.orchestra.run.smithi006.stdout: debug bluestore = 1/20 2024-09-17T23:04:38.899 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-09-17T23:04:38.899 INFO:teuthology.orchestra.run.smithi006.stdout: debug osd = 20 2024-09-17T23:04:38.899 INFO:teuthology.orchestra.run.smithi006.stdout: debug rocksdb = 4/10 2024-09-17T23:04:38.899 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd backfillfull_ratio = 0.85 2024-09-17T23:04:38.899 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd full ratio = 0.9 2024-09-17T23:04:38.899 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd nearfull ratio = 0.8 2024-09-17T23:04:38.899 INFO:teuthology.orchestra.run.smithi006.stdout: osd failsafe full ratio = 0.95 2024-09-17T23:04:38.900 INFO:teuthology.orchestra.run.smithi006.stdout: osd objectstore = bluestore 2024-09-17T23:04:38.900 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:04:38.900 INFO:teuthology.orchestra.run.smithi006.stdout:[mgr] 2024-09-17T23:04:38.900 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-09-17T23:04:38.900 INFO:teuthology.orchestra.run.smithi006.stdout: debug mgr = 20 2024-09-17T23:04:38.900 INFO:teuthology.orchestra.run.smithi006.stdout: debug mon = 20 2024-09-17T23:04:38.900 INFO:teuthology.orchestra.run.smithi006.stdout: debug auth = 20 2024-09-17T23:04:38.900 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min pgs per osd = 4 2024-09-17T23:04:38.900 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min bytes per osd = 10 2024-09-17T23:04:38.900 INFO:teuthology.orchestra.run.smithi006.stdout: mgr/telemetry/nag = false 2024-09-17T23:04:38.900 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:04:38.900 INFO:teuthology.orchestra.run.smithi006.stdout:[mon] 2024-09-17T23:04:38.900 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-09-17T23:04:38.900 INFO:teuthology.orchestra.run.smithi006.stdout: debug mon = 20 2024-09-17T23:04:38.900 INFO:teuthology.orchestra.run.smithi006.stdout: debug paxos = 20 2024-09-17T23:04:38.901 INFO:teuthology.orchestra.run.smithi006.stdout: debug auth = 20 2024-09-17T23:04:38.901 INFO:teuthology.orchestra.run.smithi006.stdout: mon data avail warn = 5 2024-09-17T23:04:38.901 INFO:teuthology.orchestra.run.smithi006.stdout: mon mgr mkfs grace = 240 2024-09-17T23:04:38.901 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min pgs per osd = 4 2024-09-17T23:04:38.901 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd reporter subtree level = osd 2024-09-17T23:04:38.901 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd prime pg temp = true 2024-09-17T23:04:38.901 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min bytes per osd = 10 2024-09-17T23:04:38.901 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:04:38.901 INFO:teuthology.orchestra.run.smithi006.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-17T23:04:38.901 INFO:teuthology.orchestra.run.smithi006.stdout: auth mon ticket ttl = 660 # 11m 2024-09-17T23:04:38.901 INFO:teuthology.orchestra.run.smithi006.stdout: auth service ticket ttl = 240 # 4m 2024-09-17T23:04:38.901 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:04:38.901 INFO:teuthology.orchestra.run.smithi006.stdout: # don't complain about insecure global_id in the test suite 2024-09-17T23:04:38.901 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-17T23:04:38.901 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-17T23:04:38.901 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:04:38.902 INFO:teuthology.orchestra.run.smithi006.stdout: # 1m isn't quite enough 2024-09-17T23:04:38.902 INFO:teuthology.orchestra.run.smithi006.stdout: mon_down_mkfs_grace = 2m 2024-09-17T23:04:38.902 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:04:38.902 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_filestore_osds = false 2024-09-17T23:04:38.902 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:04:38.902 INFO:teuthology.orchestra.run.smithi006.stdout:[client] 2024-09-17T23:04:38.902 INFO:teuthology.orchestra.run.smithi006.stdout: rgw cache enabled = true 2024-09-17T23:04:38.902 INFO:teuthology.orchestra.run.smithi006.stdout: rgw enable ops log = true 2024-09-17T23:04:38.902 INFO:teuthology.orchestra.run.smithi006.stdout: rgw enable usage log = true 2024-09-17T23:04:38.902 INFO:teuthology.orchestra.run.smithi006.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-17T23:04:38.902 INFO:teuthology.orchestra.run.smithi006.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-17T23:04:38.902 INFO:teuthology.orchestra.run.smithi006.stdout: debug rbd = 20 2024-09-17T23:04:38.902 INFO:teuthology.orchestra.run.smithi006.stdout: debug rbd_mirror = 15 2024-09-17T23:04:38.902 INFO:teuthology.orchestra.run.smithi006.stdout: log to stderr = False 2024-09-17T23:04:38.902 INFO:teuthology.orchestra.run.smithi006.stdout: rbd default features = 125 2024-09-17T23:04:38.903 INFO:teuthology.orchestra.run.smithi006.stdout: rbd mirror image policy type = none 2024-09-17T23:04:38.903 INFO:teuthology.orchestra.run.smithi006.stdout:[mon.a] 2024-09-17T23:04:38.903 INFO:teuthology.orchestra.run.smithi006.stdout:[client.mirror.0] 2024-09-17T23:04:38.903 INFO:teuthology.orchestra.run.smithi006.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-17T23:04:38.903 INFO:teuthology.orchestra.run.smithi006.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-17T23:04:38.903 INFO:teuthology.orchestra.run.smithi006.stdout:[client.mirror.1] 2024-09-17T23:04:38.903 INFO:teuthology.orchestra.run.smithi006.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-17T23:04:38.903 INFO:teuthology.orchestra.run.smithi006.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-17T23:04:38.903 INFO:teuthology.orchestra.run.smithi006.stdout:[client.mirror.2] 2024-09-17T23:04:38.903 INFO:teuthology.orchestra.run.smithi006.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-17T23:04:38.903 INFO:teuthology.orchestra.run.smithi006.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-17T23:04:38.903 INFO:teuthology.orchestra.run.smithi006.stdout:[client.mirror.3] 2024-09-17T23:04:38.903 INFO:teuthology.orchestra.run.smithi006.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-17T23:04:38.903 INFO:teuthology.orchestra.run.smithi006.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-17T23:04:38.923 INFO:teuthology.orchestra.run.smithi038.stdout:[global] 2024-09-17T23:04:38.924 INFO:teuthology.orchestra.run.smithi038.stdout: chdir = "" 2024-09-17T23:04:38.924 INFO:teuthology.orchestra.run.smithi038.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-17T23:04:38.924 INFO:teuthology.orchestra.run.smithi038.stdout: auth supported = cephx 2024-09-17T23:04:38.924 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:04:38.924 INFO:teuthology.orchestra.run.smithi038.stdout: filestore xattr use omap = true 2024-09-17T23:04:38.924 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:04:38.924 INFO:teuthology.orchestra.run.smithi038.stdout: mon clock drift allowed = 1.000 2024-09-17T23:04:38.924 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:04:38.924 INFO:teuthology.orchestra.run.smithi038.stdout: osd crush chooseleaf type = 0 2024-09-17T23:04:38.924 INFO:teuthology.orchestra.run.smithi038.stdout: auth debug = true 2024-09-17T23:04:38.924 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:04:38.924 INFO:teuthology.orchestra.run.smithi038.stdout: ms die on old message = true 2024-09-17T23:04:38.924 INFO:teuthology.orchestra.run.smithi038.stdout: ms die on bug = true 2024-09-17T23:04:38.924 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:04:38.925 INFO:teuthology.orchestra.run.smithi038.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-17T23:04:38.925 INFO:teuthology.orchestra.run.smithi038.stdout: mon pg warn max object skew = 0 2024-09-17T23:04:38.925 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:04:38.925 INFO:teuthology.orchestra.run.smithi038.stdout: # disable pg_autoscaler by default for new pools 2024-09-17T23:04:38.925 INFO:teuthology.orchestra.run.smithi038.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-17T23:04:38.925 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:04:38.925 INFO:teuthology.orchestra.run.smithi038.stdout: osd pool default size = 2 2024-09-17T23:04:38.925 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:04:38.925 INFO:teuthology.orchestra.run.smithi038.stdout: mon osd allow primary affinity = true 2024-09-17T23:04:38.925 INFO:teuthology.orchestra.run.smithi038.stdout: mon osd allow pg remap = true 2024-09-17T23:04:38.925 INFO:teuthology.orchestra.run.smithi038.stdout: mon warn on legacy crush tunables = false 2024-09-17T23:04:38.925 INFO:teuthology.orchestra.run.smithi038.stdout: mon warn on crush straw calc version zero = false 2024-09-17T23:04:38.925 INFO:teuthology.orchestra.run.smithi038.stdout: mon warn on no sortbitwise = false 2024-09-17T23:04:38.925 INFO:teuthology.orchestra.run.smithi038.stdout: mon warn on osd down out interval zero = false 2024-09-17T23:04:38.926 INFO:teuthology.orchestra.run.smithi038.stdout: mon warn on too few osds = false 2024-09-17T23:04:38.926 INFO:teuthology.orchestra.run.smithi038.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-17T23:04:38.926 INFO:teuthology.orchestra.run.smithi038.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-17T23:04:38.926 INFO:teuthology.orchestra.run.smithi038.stdout: mon_allow_pool_size_one = true 2024-09-17T23:04:38.926 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:04:38.926 INFO:teuthology.orchestra.run.smithi038.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-17T23:04:38.926 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:04:38.926 INFO:teuthology.orchestra.run.smithi038.stdout: osd default data pool replay window = 5 2024-09-17T23:04:38.926 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:04:38.926 INFO:teuthology.orchestra.run.smithi038.stdout: mon allow pool delete = true 2024-09-17T23:04:38.926 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:04:38.926 INFO:teuthology.orchestra.run.smithi038.stdout: mon cluster log file level = debug 2024-09-17T23:04:38.926 INFO:teuthology.orchestra.run.smithi038.stdout: debug asserts on shutdown = true 2024-09-17T23:04:38.926 INFO:teuthology.orchestra.run.smithi038.stdout: mon health detail to clog = false 2024-09-17T23:04:38.927 INFO:teuthology.orchestra.run.smithi038.stdout: mon host = 172.21.15.38 2024-09-17T23:04:38.927 INFO:teuthology.orchestra.run.smithi038.stdout: mon client directed command retry = 5 2024-09-17T23:04:38.927 INFO:teuthology.orchestra.run.smithi038.stdout: mon warn on pool no app = False 2024-09-17T23:04:38.927 INFO:teuthology.orchestra.run.smithi038.stdout: ms inject socket failures = 5000 2024-09-17T23:04:38.927 INFO:teuthology.orchestra.run.smithi038.stdout: fsid = 9e14216a-77a4-4aa5-9521-b021dc9cf0cd 2024-09-17T23:04:38.927 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:04:38.927 INFO:teuthology.orchestra.run.smithi038.stdout:[osd] 2024-09-17T23:04:38.927 INFO:teuthology.orchestra.run.smithi038.stdout: osd journal size = 100 2024-09-17T23:04:38.927 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:04:38.927 INFO:teuthology.orchestra.run.smithi038.stdout: osd scrub load threshold = 5.0 2024-09-17T23:04:38.927 INFO:teuthology.orchestra.run.smithi038.stdout: osd scrub max interval = 600 2024-09-17T23:04:38.927 INFO:teuthology.orchestra.run.smithi038.stdout: osd mclock profile = high_recovery_ops 2024-09-17T23:04:38.927 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:04:38.927 INFO:teuthology.orchestra.run.smithi038.stdout: osd recover clone overlap = true 2024-09-17T23:04:38.927 INFO:teuthology.orchestra.run.smithi038.stdout: osd recovery max chunk = 1048576 2024-09-17T23:04:38.927 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:04:38.928 INFO:teuthology.orchestra.run.smithi038.stdout: osd debug shutdown = true 2024-09-17T23:04:38.928 INFO:teuthology.orchestra.run.smithi038.stdout: osd debug op order = true 2024-09-17T23:04:38.928 INFO:teuthology.orchestra.run.smithi038.stdout: osd debug verify stray on activate = true 2024-09-17T23:04:38.928 INFO:teuthology.orchestra.run.smithi038.stdout: osd debug trim objects = true 2024-09-17T23:04:38.928 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:04:38.928 INFO:teuthology.orchestra.run.smithi038.stdout: osd open classes on start = true 2024-09-17T23:04:38.928 INFO:teuthology.orchestra.run.smithi038.stdout: osd debug pg log writeout = true 2024-09-17T23:04:38.928 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:04:38.928 INFO:teuthology.orchestra.run.smithi038.stdout: osd deep scrub update digest min age = 30 2024-09-17T23:04:38.928 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:04:38.928 INFO:teuthology.orchestra.run.smithi038.stdout: osd map max advance = 10 2024-09-17T23:04:38.928 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:04:38.928 INFO:teuthology.orchestra.run.smithi038.stdout: journal zero on create = true 2024-09-17T23:04:38.928 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:04:38.928 INFO:teuthology.orchestra.run.smithi038.stdout: filestore ondisk finisher threads = 3 2024-09-17T23:04:38.929 INFO:teuthology.orchestra.run.smithi038.stdout: filestore apply finisher threads = 3 2024-09-17T23:04:38.929 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:04:38.929 INFO:teuthology.orchestra.run.smithi038.stdout: bdev debug aio = true 2024-09-17T23:04:38.929 INFO:teuthology.orchestra.run.smithi038.stdout: osd debug misdirected ops = true 2024-09-17T23:04:38.929 INFO:teuthology.orchestra.run.smithi038.stdout: bluestore block size = 96636764160 2024-09-17T23:04:38.929 INFO:teuthology.orchestra.run.smithi038.stdout: bluestore compression algorithm = zstd 2024-09-17T23:04:38.929 INFO:teuthology.orchestra.run.smithi038.stdout: bluestore compression mode = aggressive 2024-09-17T23:04:38.929 INFO:teuthology.orchestra.run.smithi038.stdout: bluestore fsck on mount = True 2024-09-17T23:04:38.929 INFO:teuthology.orchestra.run.smithi038.stdout: debug bluefs = 1/20 2024-09-17T23:04:38.929 INFO:teuthology.orchestra.run.smithi038.stdout: debug bluestore = 1/20 2024-09-17T23:04:38.929 INFO:teuthology.orchestra.run.smithi038.stdout: debug ms = 1 2024-09-17T23:04:38.929 INFO:teuthology.orchestra.run.smithi038.stdout: debug osd = 20 2024-09-17T23:04:38.929 INFO:teuthology.orchestra.run.smithi038.stdout: debug rocksdb = 4/10 2024-09-17T23:04:38.929 INFO:teuthology.orchestra.run.smithi038.stdout: mon osd backfillfull_ratio = 0.85 2024-09-17T23:04:38.929 INFO:teuthology.orchestra.run.smithi038.stdout: mon osd full ratio = 0.9 2024-09-17T23:04:38.930 INFO:teuthology.orchestra.run.smithi038.stdout: mon osd nearfull ratio = 0.8 2024-09-17T23:04:38.930 INFO:teuthology.orchestra.run.smithi038.stdout: osd failsafe full ratio = 0.95 2024-09-17T23:04:38.930 INFO:teuthology.orchestra.run.smithi038.stdout: osd objectstore = bluestore 2024-09-17T23:04:38.930 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:04:38.930 INFO:teuthology.orchestra.run.smithi038.stdout:[mgr] 2024-09-17T23:04:38.930 INFO:teuthology.orchestra.run.smithi038.stdout: debug ms = 1 2024-09-17T23:04:38.930 INFO:teuthology.orchestra.run.smithi038.stdout: debug mgr = 20 2024-09-17T23:04:38.930 INFO:teuthology.orchestra.run.smithi038.stdout: debug mon = 20 2024-09-17T23:04:38.930 INFO:teuthology.orchestra.run.smithi038.stdout: debug auth = 20 2024-09-17T23:04:38.930 INFO:teuthology.orchestra.run.smithi038.stdout: mon reweight min pgs per osd = 4 2024-09-17T23:04:38.930 INFO:teuthology.orchestra.run.smithi038.stdout: mon reweight min bytes per osd = 10 2024-09-17T23:04:38.930 INFO:teuthology.orchestra.run.smithi038.stdout: mgr/telemetry/nag = false 2024-09-17T23:04:38.930 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:04:38.930 INFO:teuthology.orchestra.run.smithi038.stdout:[mon] 2024-09-17T23:04:38.930 INFO:teuthology.orchestra.run.smithi038.stdout: debug ms = 1 2024-09-17T23:04:38.931 INFO:teuthology.orchestra.run.smithi038.stdout: debug mon = 20 2024-09-17T23:04:38.931 INFO:teuthology.orchestra.run.smithi038.stdout: debug paxos = 20 2024-09-17T23:04:38.931 INFO:teuthology.orchestra.run.smithi038.stdout: debug auth = 20 2024-09-17T23:04:38.931 INFO:teuthology.orchestra.run.smithi038.stdout: mon data avail warn = 5 2024-09-17T23:04:38.931 INFO:teuthology.orchestra.run.smithi038.stdout: mon mgr mkfs grace = 240 2024-09-17T23:04:38.931 INFO:teuthology.orchestra.run.smithi038.stdout: mon reweight min pgs per osd = 4 2024-09-17T23:04:38.931 INFO:teuthology.orchestra.run.smithi038.stdout: mon osd reporter subtree level = osd 2024-09-17T23:04:38.931 INFO:teuthology.orchestra.run.smithi038.stdout: mon osd prime pg temp = true 2024-09-17T23:04:38.931 INFO:teuthology.orchestra.run.smithi038.stdout: mon reweight min bytes per osd = 10 2024-09-17T23:04:38.931 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:04:38.931 INFO:teuthology.orchestra.run.smithi038.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-17T23:04:38.931 INFO:teuthology.orchestra.run.smithi038.stdout: auth mon ticket ttl = 660 # 11m 2024-09-17T23:04:38.931 INFO:teuthology.orchestra.run.smithi038.stdout: auth service ticket ttl = 240 # 4m 2024-09-17T23:04:38.932 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:04:38.932 INFO:teuthology.orchestra.run.smithi038.stdout: # don't complain about insecure global_id in the test suite 2024-09-17T23:04:38.932 INFO:teuthology.orchestra.run.smithi038.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-17T23:04:38.932 INFO:teuthology.orchestra.run.smithi038.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-17T23:04:38.932 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:04:38.932 INFO:teuthology.orchestra.run.smithi038.stdout: # 1m isn't quite enough 2024-09-17T23:04:38.932 INFO:teuthology.orchestra.run.smithi038.stdout: mon_down_mkfs_grace = 2m 2024-09-17T23:04:38.932 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:04:38.932 INFO:teuthology.orchestra.run.smithi038.stdout: mon_warn_on_filestore_osds = false 2024-09-17T23:04:38.932 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:04:38.933 INFO:teuthology.orchestra.run.smithi038.stdout:[client] 2024-09-17T23:04:38.933 INFO:teuthology.orchestra.run.smithi038.stdout: rgw cache enabled = true 2024-09-17T23:04:38.933 INFO:teuthology.orchestra.run.smithi038.stdout: rgw enable ops log = true 2024-09-17T23:04:38.933 INFO:teuthology.orchestra.run.smithi038.stdout: rgw enable usage log = true 2024-09-17T23:04:38.933 INFO:teuthology.orchestra.run.smithi038.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-17T23:04:38.933 INFO:teuthology.orchestra.run.smithi038.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-17T23:04:38.933 INFO:teuthology.orchestra.run.smithi038.stdout: debug rbd = 20 2024-09-17T23:04:38.933 INFO:teuthology.orchestra.run.smithi038.stdout: debug rbd_mirror = 15 2024-09-17T23:04:38.933 INFO:teuthology.orchestra.run.smithi038.stdout: log to stderr = False 2024-09-17T23:04:38.933 INFO:teuthology.orchestra.run.smithi038.stdout: rbd default features = 125 2024-09-17T23:04:38.934 INFO:teuthology.orchestra.run.smithi038.stdout: rbd mirror image policy type = none 2024-09-17T23:04:38.934 INFO:teuthology.orchestra.run.smithi038.stdout:[mon.a] 2024-09-17T23:04:38.934 INFO:teuthology.orchestra.run.smithi038.stdout:[client.mirror.0] 2024-09-17T23:04:38.934 INFO:teuthology.orchestra.run.smithi038.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-17T23:04:38.934 INFO:teuthology.orchestra.run.smithi038.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-17T23:04:38.934 INFO:teuthology.orchestra.run.smithi038.stdout:[client.mirror.1] 2024-09-17T23:04:38.934 INFO:teuthology.orchestra.run.smithi038.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-17T23:04:38.934 INFO:teuthology.orchestra.run.smithi038.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-17T23:04:38.934 INFO:teuthology.orchestra.run.smithi038.stdout:[client.mirror.2] 2024-09-17T23:04:38.934 INFO:teuthology.orchestra.run.smithi038.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-17T23:04:38.934 INFO:teuthology.orchestra.run.smithi038.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-17T23:04:38.935 INFO:teuthology.orchestra.run.smithi038.stdout:[client.mirror.3] 2024-09-17T23:04:38.935 INFO:teuthology.orchestra.run.smithi038.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-17T23:04:38.935 INFO:teuthology.orchestra.run.smithi038.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-17T23:04:38.935 INFO:tasks.ceph:Creating admin key on cluster2.mon.a... 2024-09-17T23:04:38.936 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/cluster2.keyring 2024-09-17T23:04:39.031 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-17T23:04:39.032 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-09-17T23:04:39.032 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/etc/ceph/cluster2.keyring of=/dev/stdout 2024-09-17T23:04:39.047 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-09-17T23:04:39.048 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/home/ubuntu/cephtest/cluster2.monmap of=/dev/stdout 2024-09-17T23:04:39.102 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi006.front.sepia.ceph.com 2024-09-17T23:04:39.102 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-09-17T23:04:39.103 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/etc/ceph/cluster2.keyring 2024-09-17T23:04:39.103 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2024-09-17T23:04:39.145 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-09-17T23:04:39.145 DEBUG:teuthology.orchestra.run.smithi006:> dd of=/home/ubuntu/cephtest/cluster2.monmap 2024-09-17T23:04:39.203 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi038.front.sepia.ceph.com 2024-09-17T23:04:39.203 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-09-17T23:04:39.203 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/etc/ceph/cluster2.keyring 2024-09-17T23:04:39.203 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2024-09-17T23:04:39.240 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-09-17T23:04:39.240 DEBUG:teuthology.orchestra.run.smithi038:> dd of=/home/ubuntu/cephtest/cluster2.monmap 2024-09-17T23:04:39.296 INFO:tasks.ceph:Setting up mon nodes... 2024-09-17T23:04:39.296 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-17T23:04:39.296 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /var/lib/ceph/mgr/cluster2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/cluster2-x/keyring 2024-09-17T23:04:39.356 INFO:teuthology.orchestra.run.smithi006.stdout:creating /var/lib/ceph/mgr/cluster2-x/keyring 2024-09-17T23:04:39.358 INFO:tasks.ceph:Setting up mds nodes... 2024-09-17T23:04:39.359 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-17T23:04:39.359 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/cluster2.client.0.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.0.keyring 2024-09-17T23:04:39.403 INFO:teuthology.orchestra.run.smithi006.stdout:creating /etc/ceph/cluster2.client.0.keyring 2024-09-17T23:04:39.417 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror /etc/ceph/cluster2.client.mirror.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.keyring 2024-09-17T23:04:39.461 INFO:teuthology.orchestra.run.smithi038.stdout:creating /etc/ceph/cluster2.client.mirror.keyring 2024-09-17T23:04:39.473 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.0 /etc/ceph/cluster2.client.mirror.0.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.0.keyring 2024-09-17T23:04:39.559 INFO:teuthology.orchestra.run.smithi038.stdout:creating /etc/ceph/cluster2.client.mirror.0.keyring 2024-09-17T23:04:39.572 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.1 /etc/ceph/cluster2.client.mirror.1.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.1.keyring 2024-09-17T23:04:39.664 INFO:teuthology.orchestra.run.smithi038.stdout:creating /etc/ceph/cluster2.client.mirror.1.keyring 2024-09-17T23:04:39.677 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.2 /etc/ceph/cluster2.client.mirror.2.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.2.keyring 2024-09-17T23:04:39.760 INFO:teuthology.orchestra.run.smithi038.stdout:creating /etc/ceph/cluster2.client.mirror.2.keyring 2024-09-17T23:04:39.773 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.3 /etc/ceph/cluster2.client.mirror.3.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.3.keyring 2024-09-17T23:04:39.856 INFO:teuthology.orchestra.run.smithi038.stdout:creating /etc/ceph/cluster2.client.mirror.3.keyring 2024-09-17T23:04:39.869 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.4 /etc/ceph/cluster2.client.mirror.4.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.4.keyring 2024-09-17T23:04:39.951 INFO:teuthology.orchestra.run.smithi038.stdout:creating /etc/ceph/cluster2.client.mirror.4.keyring 2024-09-17T23:04:39.965 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.5 /etc/ceph/cluster2.client.mirror.5.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.5.keyring 2024-09-17T23:04:40.047 INFO:teuthology.orchestra.run.smithi038.stdout:creating /etc/ceph/cluster2.client.mirror.5.keyring 2024-09-17T23:04:40.060 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.6 /etc/ceph/cluster2.client.mirror.6.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.6.keyring 2024-09-17T23:04:40.143 INFO:teuthology.orchestra.run.smithi038.stdout:creating /etc/ceph/cluster2.client.mirror.6.keyring 2024-09-17T23:04:40.156 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-17T23:04:40.156 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi006.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi038.front.sepia.ceph.com'): {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-17T23:04:40.156 DEBUG:teuthology.orchestra.run.smithi038:> sudo mkdir -p /var/lib/ceph/osd/cluster2-0 2024-09-17T23:04:40.222 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-17T23:04:40.222 INFO:tasks.ceph:role: cluster2.osd.0 2024-09-17T23:04:40.222 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi038.front.sepia.ceph.com 2024-09-17T23:04:40.222 DEBUG:teuthology.orchestra.run.smithi038:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-17T23:04:40.316 INFO:teuthology.orchestra.run.smithi038.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-17T23:04:40.317 INFO:teuthology.orchestra.run.smithi038.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-17T23:04:40.317 INFO:teuthology.orchestra.run.smithi038.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-17T23:04:40.317 INFO:teuthology.orchestra.run.smithi038.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-17T23:04:40.317 INFO:teuthology.orchestra.run.smithi038.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-17T23:04:40.317 INFO:teuthology.orchestra.run.smithi038.stdout: = sunit=0 swidth=0 blks 2024-09-17T23:04:40.317 INFO:teuthology.orchestra.run.smithi038.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-17T23:04:40.317 INFO:teuthology.orchestra.run.smithi038.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-17T23:04:40.317 INFO:teuthology.orchestra.run.smithi038.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-17T23:04:40.317 INFO:teuthology.orchestra.run.smithi038.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-17T23:04:41.655 INFO:teuthology.orchestra.run.smithi038.stdout:Discarding blocks...Done. 2024-09-17T23:04:41.658 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi038.front.sepia.ceph.com -o noatime 2024-09-17T23:04:41.658 DEBUG:teuthology.orchestra.run.smithi038:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/cluster2-0 2024-09-17T23:04:41.700 DEBUG:teuthology.orchestra.run.smithi038:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-0 2024-09-17T23:04:41.776 DEBUG:teuthology.orchestra.run.smithi038:> sudo mkdir -p /var/lib/ceph/osd/cluster2-1 2024-09-17T23:04:41.844 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-17T23:04:41.844 INFO:tasks.ceph:role: cluster2.osd.1 2024-09-17T23:04:41.845 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi038.front.sepia.ceph.com 2024-09-17T23:04:41.845 DEBUG:teuthology.orchestra.run.smithi038:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-17T23:04:41.942 INFO:teuthology.orchestra.run.smithi038.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-17T23:04:41.942 INFO:teuthology.orchestra.run.smithi038.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-17T23:04:41.942 INFO:teuthology.orchestra.run.smithi038.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-17T23:04:41.942 INFO:teuthology.orchestra.run.smithi038.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-17T23:04:41.942 INFO:teuthology.orchestra.run.smithi038.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-17T23:04:41.942 INFO:teuthology.orchestra.run.smithi038.stdout: = sunit=0 swidth=0 blks 2024-09-17T23:04:41.942 INFO:teuthology.orchestra.run.smithi038.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-17T23:04:41.942 INFO:teuthology.orchestra.run.smithi038.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-17T23:04:41.942 INFO:teuthology.orchestra.run.smithi038.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-17T23:04:41.943 INFO:teuthology.orchestra.run.smithi038.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-17T23:04:43.260 INFO:teuthology.orchestra.run.smithi038.stdout:Discarding blocks...Done. 2024-09-17T23:04:43.262 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi038.front.sepia.ceph.com -o noatime 2024-09-17T23:04:43.263 DEBUG:teuthology.orchestra.run.smithi038:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/cluster2-1 2024-09-17T23:04:43.303 DEBUG:teuthology.orchestra.run.smithi038:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-1 2024-09-17T23:04:43.376 DEBUG:teuthology.orchestra.run.smithi038:> sudo mkdir -p /var/lib/ceph/osd/cluster2-2 2024-09-17T23:04:43.443 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-17T23:04:43.443 INFO:tasks.ceph:role: cluster2.osd.2 2024-09-17T23:04:43.443 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi038.front.sepia.ceph.com 2024-09-17T23:04:43.443 DEBUG:teuthology.orchestra.run.smithi038:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-17T23:04:43.540 INFO:teuthology.orchestra.run.smithi038.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-17T23:04:43.540 INFO:teuthology.orchestra.run.smithi038.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-17T23:04:43.541 INFO:teuthology.orchestra.run.smithi038.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-17T23:04:43.541 INFO:teuthology.orchestra.run.smithi038.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-17T23:04:43.541 INFO:teuthology.orchestra.run.smithi038.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-17T23:04:43.541 INFO:teuthology.orchestra.run.smithi038.stdout: = sunit=0 swidth=0 blks 2024-09-17T23:04:43.541 INFO:teuthology.orchestra.run.smithi038.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-17T23:04:43.541 INFO:teuthology.orchestra.run.smithi038.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-17T23:04:43.541 INFO:teuthology.orchestra.run.smithi038.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-17T23:04:43.541 INFO:teuthology.orchestra.run.smithi038.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-17T23:04:44.855 INFO:teuthology.orchestra.run.smithi038.stdout:Discarding blocks...Done. 2024-09-17T23:04:44.858 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi038.front.sepia.ceph.com -o noatime 2024-09-17T23:04:44.858 DEBUG:teuthology.orchestra.run.smithi038:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/cluster2-2 2024-09-17T23:04:44.895 DEBUG:teuthology.orchestra.run.smithi038:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-2 2024-09-17T23:04:44.969 DEBUG:teuthology.orchestra.run.smithi038:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/cluster2.monmap 2024-09-17T23:04:45.070 INFO:teuthology.orchestra.run.smithi038.stderr:2024-09-17T23:04:45.068+0000 7f663dc13640 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-0/keyring: can't open /var/lib/ceph/osd/cluster2-0/keyring: (2) No such file or directory 2024-09-17T23:04:45.071 INFO:teuthology.orchestra.run.smithi038.stderr:2024-09-17T23:04:45.068+0000 7f663dc13640 -1 created new key in keyring /var/lib/ceph/osd/cluster2-0/keyring 2024-09-17T23:04:45.071 INFO:teuthology.orchestra.run.smithi038.stderr:2024-09-17T23:04:45.068+0000 7f663dc13640 -1 bdev(0x55c59a5e3800 /var/lib/ceph/osd/cluster2-0/block) open stat got: (1) Operation not permitted 2024-09-17T23:04:45.071 INFO:teuthology.orchestra.run.smithi038.stderr:2024-09-17T23:04:45.068+0000 7f663dc13640 -1 bluestore(/var/lib/ceph/osd/cluster2-0) _read_fsid unparsable uuid 2024-09-17T23:04:48.001 DEBUG:teuthology.orchestra.run.smithi038:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-0 2024-09-17T23:04:48.032 DEBUG:teuthology.orchestra.run.smithi038:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/cluster2.monmap 2024-09-17T23:04:48.125 INFO:teuthology.orchestra.run.smithi038.stderr:2024-09-17T23:04:48.123+0000 7fef4f0a5640 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-1/keyring: can't open /var/lib/ceph/osd/cluster2-1/keyring: (2) No such file or directory 2024-09-17T23:04:48.126 INFO:teuthology.orchestra.run.smithi038.stderr:2024-09-17T23:04:48.123+0000 7fef4f0a5640 -1 created new key in keyring /var/lib/ceph/osd/cluster2-1/keyring 2024-09-17T23:04:48.126 INFO:teuthology.orchestra.run.smithi038.stderr:2024-09-17T23:04:48.123+0000 7fef4f0a5640 -1 bdev(0x562bc4b87800 /var/lib/ceph/osd/cluster2-1/block) open stat got: (1) Operation not permitted 2024-09-17T23:04:48.126 INFO:teuthology.orchestra.run.smithi038.stderr:2024-09-17T23:04:48.123+0000 7fef4f0a5640 -1 bluestore(/var/lib/ceph/osd/cluster2-1) _read_fsid unparsable uuid 2024-09-17T23:04:51.296 DEBUG:teuthology.orchestra.run.smithi038:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-1 2024-09-17T23:04:51.327 DEBUG:teuthology.orchestra.run.smithi038:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/cluster2.monmap 2024-09-17T23:04:51.420 INFO:teuthology.orchestra.run.smithi038.stderr:2024-09-17T23:04:51.417+0000 7f9029e0d640 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-2/keyring: can't open /var/lib/ceph/osd/cluster2-2/keyring: (2) No such file or directory 2024-09-17T23:04:51.420 INFO:teuthology.orchestra.run.smithi038.stderr:2024-09-17T23:04:51.417+0000 7f9029e0d640 -1 created new key in keyring /var/lib/ceph/osd/cluster2-2/keyring 2024-09-17T23:04:51.420 INFO:teuthology.orchestra.run.smithi038.stderr:2024-09-17T23:04:51.417+0000 7f9029e0d640 -1 bdev(0x5604103bd800 /var/lib/ceph/osd/cluster2-2/block) open stat got: (1) Operation not permitted 2024-09-17T23:04:51.420 INFO:teuthology.orchestra.run.smithi038.stderr:2024-09-17T23:04:51.417+0000 7f9029e0d640 -1 bluestore(/var/lib/ceph/osd/cluster2-2) _read_fsid unparsable uuid 2024-09-17T23:04:55.098 DEBUG:teuthology.orchestra.run.smithi038:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-2 2024-09-17T23:04:55.128 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-17T23:04:55.129 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-09-17T23:04:55.129 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/mgr/cluster2-x/keyring of=/dev/stdout 2024-09-17T23:04:55.160 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-09-17T23:04:55.160 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/var/lib/ceph/osd/cluster2-0/keyring of=/dev/stdout 2024-09-17T23:04:55.198 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-09-17T23:04:55.199 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/var/lib/ceph/osd/cluster2-1/keyring of=/dev/stdout 2024-09-17T23:04:55.266 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-09-17T23:04:55.266 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/var/lib/ceph/osd/cluster2-2/keyring of=/dev/stdout 2024-09-17T23:04:55.335 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-09-17T23:04:55.335 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/etc/ceph/cluster2.client.0.keyring of=/dev/stdout 2024-09-17T23:04:55.354 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-09-17T23:04:55.354 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/etc/ceph/cluster2.client.mirror.keyring of=/dev/stdout 2024-09-17T23:04:55.393 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-09-17T23:04:55.393 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/etc/ceph/cluster2.client.mirror.0.keyring of=/dev/stdout 2024-09-17T23:04:55.449 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-09-17T23:04:55.449 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/etc/ceph/cluster2.client.mirror.1.keyring of=/dev/stdout 2024-09-17T23:04:55.506 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-09-17T23:04:55.506 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/etc/ceph/cluster2.client.mirror.2.keyring of=/dev/stdout 2024-09-17T23:04:55.563 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-09-17T23:04:55.563 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/etc/ceph/cluster2.client.mirror.3.keyring of=/dev/stdout 2024-09-17T23:04:55.620 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-09-17T23:04:55.620 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/etc/ceph/cluster2.client.mirror.4.keyring of=/dev/stdout 2024-09-17T23:04:55.677 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-09-17T23:04:55.677 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/etc/ceph/cluster2.client.mirror.5.keyring of=/dev/stdout 2024-09-17T23:04:55.733 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-09-17T23:04:55.734 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/etc/ceph/cluster2.client.mirror.6.keyring of=/dev/stdout 2024-09-17T23:04:55.789 INFO:tasks.ceph:Adding keys to all mons... 2024-09-17T23:04:55.790 DEBUG:teuthology.orchestra.run.smithi038:> sudo tee -a /etc/ceph/cluster2.keyring 2024-09-17T23:04:55.853 INFO:teuthology.orchestra.run.smithi038.stdout:[mgr.x] 2024-09-17T23:04:55.854 INFO:teuthology.orchestra.run.smithi038.stdout: key = AQAHC+pmF+glFRAAZFnoqU6JDrBjZlmHa3b4Aw== 2024-09-17T23:04:55.854 INFO:teuthology.orchestra.run.smithi038.stdout:[osd.0] 2024-09-17T23:04:55.854 INFO:teuthology.orchestra.run.smithi038.stdout: key = AQANC+pmO7MhBBAAuzXODsMm33AjjoTuTpeO/Q== 2024-09-17T23:04:55.854 INFO:teuthology.orchestra.run.smithi038.stdout:[osd.1] 2024-09-17T23:04:55.854 INFO:teuthology.orchestra.run.smithi038.stdout: key = AQAQC+pmB7VlBxAAHGP0iOWkba+oFA2ezAUpbg== 2024-09-17T23:04:55.854 INFO:teuthology.orchestra.run.smithi038.stdout:[osd.2] 2024-09-17T23:04:55.854 INFO:teuthology.orchestra.run.smithi038.stdout: key = AQATC+pmjg7vGBAAhz29dbDfwAAEPnJYREEEzQ== 2024-09-17T23:04:55.854 INFO:teuthology.orchestra.run.smithi038.stdout:[client.0] 2024-09-17T23:04:55.854 INFO:teuthology.orchestra.run.smithi038.stdout: key = AQAHC+pmK/n8FxAA1r8LS0nnOcbApbnFOcGang== 2024-09-17T23:04:55.854 INFO:teuthology.orchestra.run.smithi038.stdout:[client.mirror] 2024-09-17T23:04:55.854 INFO:teuthology.orchestra.run.smithi038.stdout: key = AQAHC+pmTdhiGxAAft2fvH1QDiXNxRJwcJLNpw== 2024-09-17T23:04:55.854 INFO:teuthology.orchestra.run.smithi038.stdout:[client.mirror.0] 2024-09-17T23:04:55.854 INFO:teuthology.orchestra.run.smithi038.stdout: key = AQAHC+pmEzQ4IRAA0pSkPl5sElORKwl87e1NCQ== 2024-09-17T23:04:55.855 INFO:teuthology.orchestra.run.smithi038.stdout:[client.mirror.1] 2024-09-17T23:04:55.855 INFO:teuthology.orchestra.run.smithi038.stdout: key = AQAHC+pmdRaAJxAAnYqUAXNFzNwm/Ni/eIMMFw== 2024-09-17T23:04:55.855 INFO:teuthology.orchestra.run.smithi038.stdout:[client.mirror.2] 2024-09-17T23:04:55.855 INFO:teuthology.orchestra.run.smithi038.stdout: key = AQAHC+pmUUU4LRAAJQ97LEyaJosHdgSb5DXn4Q== 2024-09-17T23:04:55.855 INFO:teuthology.orchestra.run.smithi038.stdout:[client.mirror.3] 2024-09-17T23:04:55.855 INFO:teuthology.orchestra.run.smithi038.stdout: key = AQAHC+pmFiXxMhAA207RAyAmrhRt5+u0Nkv/rw== 2024-09-17T23:04:55.855 INFO:teuthology.orchestra.run.smithi038.stdout:[client.mirror.4] 2024-09-17T23:04:55.855 INFO:teuthology.orchestra.run.smithi038.stdout: key = AQAHC+pmLeKlOBAAyIr8wYz5R8rTjxqrMePOdw== 2024-09-17T23:04:55.855 INFO:teuthology.orchestra.run.smithi038.stdout:[client.mirror.5] 2024-09-17T23:04:55.855 INFO:teuthology.orchestra.run.smithi038.stdout: key = AQAIC+pm/TfCAhAARt3SAxKActbHQnYDSOn3ag== 2024-09-17T23:04:55.855 INFO:teuthology.orchestra.run.smithi038.stdout:[client.mirror.6] 2024-09-17T23:04:55.855 INFO:teuthology.orchestra.run.smithi038.stdout: key = AQAIC+pmZZR3CBAAD5IDS3BNr1QNP2mUpg4aJQ== 2024-09-17T23:04:55.856 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-17T23:04:55.949 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T23:04:56.001 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T23:04:56.054 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T23:04:56.106 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-17T23:04:56.161 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-17T23:04:56.216 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-17T23:04:56.271 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-17T23:04:56.326 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-17T23:04:56.381 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-17T23:04:56.436 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.4 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-17T23:04:56.491 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.5 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-17T23:04:56.546 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.6 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-17T23:04:56.602 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-17T23:04:56.602 DEBUG:teuthology.orchestra.run.smithi038:> sudo mkdir -p /var/lib/ceph/mon/cluster2-a 2024-09-17T23:04:56.627 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster cluster2 --mkfs -i a --monmap /home/ubuntu/cephtest/cluster2.monmap --keyring /etc/ceph/cluster2.keyring 2024-09-17T23:04:56.739 DEBUG:teuthology.orchestra.run.smithi038:> sudo chown -R ceph:ceph /var/lib/ceph/mon/cluster2-a 2024-09-17T23:04:56.765 DEBUG:teuthology.orchestra.run.smithi038:> rm -- /home/ubuntu/cephtest/cluster2.monmap 2024-09-17T23:04:56.821 INFO:tasks.ceph:Starting mon daemons in cluster cluster2... 2024-09-17T23:04:56.821 INFO:tasks.ceph.cluster2.mon.a:Restarting daemon 2024-09-17T23:04:56.821 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster cluster2 -i a 2024-09-17T23:04:56.863 INFO:tasks.ceph.cluster2.mon.a:Started 2024-09-17T23:04:56.864 INFO:tasks.ceph:Starting mgr daemons in cluster cluster2... 2024-09-17T23:04:56.864 INFO:tasks.ceph.cluster2.mgr.x:Restarting daemon 2024-09-17T23:04:56.864 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster cluster2 -i x 2024-09-17T23:04:56.867 INFO:tasks.ceph.cluster2.mgr.x:Started 2024-09-17T23:04:56.867 INFO:tasks.ceph:Setting crush tunables to default 2024-09-17T23:04:56.867 DEBUG:teuthology.orchestra.run.smithi038:> sudo ceph --cluster cluster2 osd crush tunables default 2024-09-17T23:04:57.058 INFO:teuthology.orchestra.run.smithi038.stderr:adjusted tunables profile to default 2024-09-17T23:04:57.068 INFO:tasks.ceph:check_enable_crimson: False 2024-09-17T23:04:57.069 INFO:tasks.ceph:Starting osd daemons in cluster cluster2... 2024-09-17T23:04:57.069 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-09-17T23:04:57.069 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/var/lib/ceph/osd/cluster2-0/fsid of=/dev/stdout 2024-09-17T23:04:57.069 INFO:tasks.ceph.cluster2.mgr.x.smithi006.stderr:2024-09-17T23:04:57.066+0000 7fb314799040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-17T23:04:57.094 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-09-17T23:04:57.094 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/var/lib/ceph/osd/cluster2-1/fsid of=/dev/stdout 2024-09-17T23:04:57.112 INFO:tasks.ceph.cluster2.mgr.x.smithi006.stderr:2024-09-17T23:04:57.109+0000 7fb314799040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-17T23:04:57.159 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-09-17T23:04:57.160 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/var/lib/ceph/osd/cluster2-2/fsid of=/dev/stdout 2024-09-17T23:04:57.225 DEBUG:teuthology.orchestra.run.smithi038:> sudo ceph --cluster cluster2 osd new 411d4107-4e3a-4b68-a515-eafd1b3b3b4f 0 2024-09-17T23:04:57.249 INFO:tasks.ceph.cluster2.mgr.x.smithi006.stderr:2024-09-17T23:04:57.247+0000 7fb314799040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-17T23:04:57.300 INFO:tasks.ceph.cluster2.mgr.x.smithi006.stderr:2024-09-17T23:04:57.298+0000 7fb314799040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-17T23:04:57.414 INFO:tasks.ceph.cluster2.mgr.x.smithi006.stderr:2024-09-17T23:04:57.412+0000 7fb314799040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-17T23:04:57.445 INFO:teuthology.orchestra.run.smithi038.stdout:0 2024-09-17T23:04:57.456 DEBUG:teuthology.orchestra.run.smithi038:> sudo ceph --cluster cluster2 osd new 7a899e9e-c234-4128-8b3c-5234e7a66aed 1 2024-09-17T23:04:57.512 INFO:tasks.ceph.cluster2.mgr.x.smithi006.stderr:2024-09-17T23:04:57.510+0000 7fb314799040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-17T23:04:57.636 INFO:teuthology.orchestra.run.smithi038.stdout:1 2024-09-17T23:04:57.647 DEBUG:teuthology.orchestra.run.smithi038:> sudo ceph --cluster cluster2 osd new 06a70fc4-b466-44d3-9800-2e0c9780993e 2 2024-09-17T23:04:57.790 INFO:tasks.ceph.cluster2.mgr.x.smithi006.stderr:2024-09-17T23:04:57.787+0000 7fb314799040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-17T23:04:57.833 INFO:teuthology.orchestra.run.smithi038.stdout:2 2024-09-17T23:04:57.836 INFO:tasks.ceph.cluster2.mgr.x.smithi006.stderr:2024-09-17T23:04:57.833+0000 7fb314799040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-17T23:04:57.844 INFO:tasks.ceph.cluster2.osd.0:Restarting daemon 2024-09-17T23:04:57.844 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 0 2024-09-17T23:04:57.846 INFO:tasks.ceph.cluster2.osd.0:Started 2024-09-17T23:04:57.846 INFO:tasks.ceph.cluster2.osd.1:Restarting daemon 2024-09-17T23:04:57.846 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 1 2024-09-17T23:04:57.848 INFO:tasks.ceph.cluster2.osd.1:Started 2024-09-17T23:04:57.848 INFO:tasks.ceph.cluster2.osd.2:Restarting daemon 2024-09-17T23:04:57.848 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 2 2024-09-17T23:04:57.850 INFO:tasks.ceph.cluster2.osd.2:Started 2024-09-17T23:04:57.851 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-09-17T23:04:57.958 INFO:tasks.ceph.cluster2.mgr.x.smithi006.stderr:2024-09-17T23:04:57.956+0000 7fb314799040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-17T23:04:58.025 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:04:58.025 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":5,"fsid":"9e14216a-77a4-4aa5-9521-b021dc9cf0cd","created":"2024-09-17T23:04:56.974990+0000","modified":"2024-09-17T23:04:57.825666+0000","last_up_change":"0.000000","last_in_change":"2024-09-17T23:04:57.825666+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"411d4107-4e3a-4b68-a515-eafd1b3b3b4f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"7a899e9e-c234-4128-8b3c-5234e7a66aed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"06a70fc4-b466-44d3-9800-2e0c9780993e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T23:04:58.036 INFO:tasks.ceph.ceph_manager.cluster2:[] 2024-09-17T23:04:58.036 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-17T23:04:58.338 DEBUG:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2024-09-17T23:04:58.500 INFO:teuthology.misc.health.smithi038.stdout: 2024-09-17T23:04:58.500 INFO:teuthology.misc.health.smithi038.stdout:{"epoch":5,"fsid":"9e14216a-77a4-4aa5-9521-b021dc9cf0cd","created":"2024-09-17T23:04:56.974990+0000","modified":"2024-09-17T23:04:57.825666+0000","last_up_change":"0.000000","last_in_change":"2024-09-17T23:04:57.825666+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"411d4107-4e3a-4b68-a515-eafd1b3b3b4f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"7a899e9e-c234-4128-8b3c-5234e7a66aed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"06a70fc4-b466-44d3-9800-2e0c9780993e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T23:04:58.509 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-09-17T23:04:58.696 INFO:tasks.ceph.cluster2.mgr.x.smithi006.stderr:2024-09-17T23:04:58.694+0000 7fb314799040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-17T23:04:58.742 INFO:tasks.ceph.cluster2.mgr.x.smithi006.stderr:2024-09-17T23:04:58.740+0000 7fb314799040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-17T23:04:59.076 INFO:tasks.ceph.cluster2.mgr.x.smithi006.stderr:2024-09-17T23:04:59.074+0000 7fb314799040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-17T23:04:59.164 INFO:tasks.ceph.cluster2.mgr.x.smithi006.stderr:2024-09-17T23:04:59.161+0000 7fb314799040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-17T23:04:59.272 INFO:tasks.ceph.cluster2.osd.2.smithi038.stderr:2024-09-17T23:04:59.269+0000 7f1c5be0c640 -1 Falling back to public interface 2024-09-17T23:04:59.276 INFO:tasks.ceph.cluster2.osd.1.smithi038.stderr:2024-09-17T23:04:59.273+0000 7f055c189640 -1 Falling back to public interface 2024-09-17T23:04:59.276 INFO:tasks.ceph.cluster2.mgr.x.smithi006.stderr:2024-09-17T23:04:59.274+0000 7fb314799040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-17T23:04:59.281 INFO:tasks.ceph.cluster2.osd.0.smithi038.stderr:2024-09-17T23:04:59.278+0000 7fa27ae1e640 -1 Falling back to public interface 2024-09-17T23:04:59.321 INFO:tasks.ceph.cluster2.mgr.x.smithi006.stderr:2024-09-17T23:04:59.319+0000 7fb314799040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-17T23:04:59.438 INFO:tasks.ceph.cluster2.mgr.x.smithi006.stderr:2024-09-17T23:04:59.435+0000 7fb314799040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-17T23:04:59.492 INFO:tasks.ceph.cluster2.mgr.x.smithi006.stderr:2024-09-17T23:04:59.490+0000 7fb314799040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-17T23:04:59.536 INFO:tasks.ceph.cluster2.mgr.x.smithi006.stderr:2024-09-17T23:04:59.534+0000 7fb314799040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-17T23:04:59.654 INFO:tasks.ceph.cluster2.mgr.x.smithi006.stderr:2024-09-17T23:04:59.652+0000 7fb314799040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-17T23:04:59.706 INFO:tasks.ceph.cluster2.mgr.x.smithi006.stderr:2024-09-17T23:04:59.704+0000 7fb314799040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-17T23:04:59.756 INFO:tasks.ceph.cluster2.mgr.x.smithi006.stderr:2024-09-17T23:04:59.754+0000 7fb314799040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-17T23:04:59.812 INFO:tasks.ceph.cluster2.mgr.x.smithi006.stderr:2024-09-17T23:04:59.809+0000 7fb314799040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-17T23:04:59.856 INFO:tasks.ceph.cluster2.mgr.x.smithi006.stderr:2024-09-17T23:04:59.854+0000 7fb314799040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-17T23:04:59.900 INFO:tasks.ceph.cluster2.mgr.x.smithi006.stderr:2024-09-17T23:04:59.898+0000 7fb314799040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-17T23:05:00.497 INFO:tasks.ceph.cluster2.mgr.x.smithi006.stderr:2024-09-17T23:05:00.495+0000 7fb314799040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-17T23:05:01.515 INFO:tasks.ceph.cluster2.mgr.x.smithi006.stderr:2024-09-17T23:05:01.513+0000 7fb2b50a7640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-09-17T23:05:03.335 INFO:tasks.ceph.cluster2.osd.1.smithi038.stderr:2024-09-17T23:05:03.333+0000 7f055c189640 -1 osd.1 0 log_to_monitors true 2024-09-17T23:05:03.343 INFO:tasks.ceph.cluster2.osd.2.smithi038.stderr:2024-09-17T23:05:03.341+0000 7f1c5be0c640 -1 osd.2 0 log_to_monitors true 2024-09-17T23:05:03.595 INFO:tasks.ceph.cluster2.osd.0.smithi038.stderr:2024-09-17T23:05:03.593+0000 7fa27ae1e640 -1 osd.0 0 log_to_monitors true 2024-09-17T23:05:04.812 DEBUG:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2024-09-17T23:05:05.310 INFO:teuthology.misc.health.smithi038.stdout: 2024-09-17T23:05:05.310 INFO:teuthology.misc.health.smithi038.stdout:{"epoch":7,"fsid":"9e14216a-77a4-4aa5-9521-b021dc9cf0cd","created":"2024-09-17T23:04:56.974990+0000","modified":"2024-09-17T23:05:04.527400+0000","last_up_change":"0.000000","last_in_change":"2024-09-17T23:04:57.825666+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"411d4107-4e3a-4b68-a515-eafd1b3b3b4f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"7a899e9e-c234-4128-8b3c-5234e7a66aed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"06a70fc4-b466-44d3-9800-2e0c9780993e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T23:05:05.325 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-09-17T23:05:05.526 INFO:tasks.ceph.cluster2.osd.2.smithi038.stderr:2024-09-17T23:05:05.523+0000 7f1c576c9640 -1 osd.2 0 waiting for initial osdmap 2024-09-17T23:05:05.538 INFO:tasks.ceph.cluster2.osd.2.smithi038.stderr:2024-09-17T23:05:05.536+0000 7f1c4f4e9640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T23:05:05.549 INFO:tasks.ceph.cluster2.osd.1.smithi038.stderr:2024-09-17T23:05:05.547+0000 7f055892c640 -1 osd.1 0 waiting for initial osdmap 2024-09-17T23:05:05.564 INFO:tasks.ceph.cluster2.osd.1.smithi038.stderr:2024-09-17T23:05:05.562+0000 7f054ff38640 -1 osd.1 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T23:05:06.314 INFO:tasks.ceph.cluster2.osd.0.smithi038.stderr:2024-09-17T23:05:06.311+0000 7fa2775c1640 -1 osd.0 0 waiting for initial osdmap 2024-09-17T23:05:06.323 INFO:tasks.ceph.cluster2.osd.0.smithi038.stderr:2024-09-17T23:05:06.321+0000 7fa26ebcd640 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T23:05:11.628 DEBUG:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2024-09-17T23:05:11.922 INFO:teuthology.misc.health.smithi038.stdout: 2024-09-17T23:05:11.922 INFO:teuthology.misc.health.smithi038.stdout:{"epoch":13,"fsid":"9e14216a-77a4-4aa5-9521-b021dc9cf0cd","created":"2024-09-17T23:04:56.974990+0000","modified":"2024-09-17T23:05:10.574464+0000","last_up_change":"2024-09-17T23:05:06.530460+0000","last_in_change":"2024-09-17T23:04:57.825666+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T23:05:08.551128+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"411d4107-4e3a-4b68-a515-eafd1b3b3b4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":1291026539},{"type":"v1","addr":"172.21.15.38:6817","nonce":1291026539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":1291026539},{"type":"v1","addr":"172.21.15.38:6819","nonce":1291026539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":1291026539},{"type":"v1","addr":"172.21.15.38:6823","nonce":1291026539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":1291026539},{"type":"v1","addr":"172.21.15.38:6821","nonce":1291026539}]},"public_addr":"172.21.15.38:6817/1291026539","cluster_addr":"172.21.15.38:6819/1291026539","heartbeat_back_addr":"172.21.15.38:6823/1291026539","heartbeat_front_addr":"172.21.15.38:6821/1291026539","state":["exists","up"]},{"osd":1,"uuid":"7a899e9e-c234-4128-8b3c-5234e7a66aed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":2141962341},{"type":"v1","addr":"172.21.15.38:6809","nonce":2141962341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":2141962341},{"type":"v1","addr":"172.21.15.38:6811","nonce":2141962341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":2141962341},{"type":"v1","addr":"172.21.15.38:6815","nonce":2141962341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2141962341},{"type":"v1","addr":"172.21.15.38:6813","nonce":2141962341}]},"public_addr":"172.21.15.38:6809/2141962341","cluster_addr":"172.21.15.38:6811/2141962341","heartbeat_back_addr":"172.21.15.38:6815/2141962341","heartbeat_front_addr":"172.21.15.38:6813/2141962341","state":["exists","up"]},{"osd":2,"uuid":"06a70fc4-b466-44d3-9800-2e0c9780993e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":2430931791},{"type":"v1","addr":"172.21.15.38:6801","nonce":2430931791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":2430931791},{"type":"v1","addr":"172.21.15.38:6803","nonce":2430931791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2430931791},{"type":"v1","addr":"172.21.15.38:6807","nonce":2430931791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":2430931791},{"type":"v1","addr":"172.21.15.38:6805","nonce":2430931791}]},"public_addr":"172.21.15.38:6801/2430931791","cluster_addr":"172.21.15.38:6803/2430931791","heartbeat_back_addr":"172.21.15.38:6807/2430931791","heartbeat_front_addr":"172.21.15.38:6805/2430931791","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T23:05:04.623873+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T23:05:04.361551+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T23:05:04.333856+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T23:05:11.933 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-09-17T23:05:11.933 INFO:tasks.ceph:Creating RBD pool 2024-09-17T23:05:11.933 DEBUG:teuthology.orchestra.run.smithi038:> sudo ceph --cluster cluster2 osd pool create rbd 8 2024-09-17T23:05:12.592 INFO:teuthology.orchestra.run.smithi038.stderr:pool 'rbd' created 2024-09-17T23:05:12.607 DEBUG:teuthology.orchestra.run.smithi038:> rbd --cluster cluster2 pool init rbd 2024-09-17T23:05:15.624 INFO:tasks.ceph:Starting mds daemons in cluster cluster2... 2024-09-17T23:05:15.624 INFO:tasks.ceph:Waiting until cluster2 daemons up and pgs clean... 2024-09-17T23:05:15.625 INFO:tasks.ceph.ceph_manager.cluster2:waiting for mgr available 2024-09-17T23:05:15.625 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 mgr dump --format=json 2024-09-17T23:05:15.625 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-17T23:05:15.972 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:05:15.983 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":5,"flags":0,"active_gid":4099,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":248342279},{"type":"v1","addr":"172.21.15.6:6827","nonce":248342279}]},"active_addr":"172.21.15.6:6827/248342279","active_change":"2024-09-17T23:05:00.497357+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.6:0","nonce":2633472570}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.6:0","nonce":1818016226}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.6:0","nonce":2279883784}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.6:0","nonce":2866591937}]}]} 2024-09-17T23:05:15.985 INFO:tasks.ceph.ceph_manager.cluster2:mgr available! 2024-09-17T23:05:15.985 INFO:tasks.ceph.ceph_manager.cluster2:waiting for all up 2024-09-17T23:05:15.985 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-09-17T23:05:16.272 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:05:16.273 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":17,"fsid":"9e14216a-77a4-4aa5-9521-b021dc9cf0cd","created":"2024-09-17T23:04:56.974990+0000","modified":"2024-09-17T23:05:15.606877+0000","last_up_change":"2024-09-17T23:05:06.530460+0000","last_in_change":"2024-09-17T23:04:57.825666+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T23:05:08.551128+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-17T23:05:12.248805+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":17,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"411d4107-4e3a-4b68-a515-eafd1b3b3b4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":1291026539},{"type":"v1","addr":"172.21.15.38:6817","nonce":1291026539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":1291026539},{"type":"v1","addr":"172.21.15.38:6819","nonce":1291026539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":1291026539},{"type":"v1","addr":"172.21.15.38:6823","nonce":1291026539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":1291026539},{"type":"v1","addr":"172.21.15.38:6821","nonce":1291026539}]},"public_addr":"172.21.15.38:6817/1291026539","cluster_addr":"172.21.15.38:6819/1291026539","heartbeat_back_addr":"172.21.15.38:6823/1291026539","heartbeat_front_addr":"172.21.15.38:6821/1291026539","state":["exists","up"]},{"osd":1,"uuid":"7a899e9e-c234-4128-8b3c-5234e7a66aed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":2141962341},{"type":"v1","addr":"172.21.15.38:6809","nonce":2141962341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":2141962341},{"type":"v1","addr":"172.21.15.38:6811","nonce":2141962341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":2141962341},{"type":"v1","addr":"172.21.15.38:6815","nonce":2141962341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2141962341},{"type":"v1","addr":"172.21.15.38:6813","nonce":2141962341}]},"public_addr":"172.21.15.38:6809/2141962341","cluster_addr":"172.21.15.38:6811/2141962341","heartbeat_back_addr":"172.21.15.38:6815/2141962341","heartbeat_front_addr":"172.21.15.38:6813/2141962341","state":["exists","up"]},{"osd":2,"uuid":"06a70fc4-b466-44d3-9800-2e0c9780993e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":2430931791},{"type":"v1","addr":"172.21.15.38:6801","nonce":2430931791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":2430931791},{"type":"v1","addr":"172.21.15.38:6803","nonce":2430931791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2430931791},{"type":"v1","addr":"172.21.15.38:6807","nonce":2430931791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":2430931791},{"type":"v1","addr":"172.21.15.38:6805","nonce":2430931791}]},"public_addr":"172.21.15.38:6801/2430931791","cluster_addr":"172.21.15.38:6803/2430931791","heartbeat_back_addr":"172.21.15.38:6807/2430931791","heartbeat_front_addr":"172.21.15.38:6805/2430931791","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T23:05:04.623873+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T23:05:04.361551+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T23:05:04.333856+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T23:05:16.285 INFO:tasks.ceph.ceph_manager.cluster2:all up! 2024-09-17T23:05:16.285 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-09-17T23:05:16.575 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:05:16.575 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":17,"fsid":"9e14216a-77a4-4aa5-9521-b021dc9cf0cd","created":"2024-09-17T23:04:56.974990+0000","modified":"2024-09-17T23:05:15.606877+0000","last_up_change":"2024-09-17T23:05:06.530460+0000","last_in_change":"2024-09-17T23:04:57.825666+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T23:05:08.551128+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-17T23:05:12.248805+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":17,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"411d4107-4e3a-4b68-a515-eafd1b3b3b4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":1291026539},{"type":"v1","addr":"172.21.15.38:6817","nonce":1291026539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":1291026539},{"type":"v1","addr":"172.21.15.38:6819","nonce":1291026539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":1291026539},{"type":"v1","addr":"172.21.15.38:6823","nonce":1291026539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":1291026539},{"type":"v1","addr":"172.21.15.38:6821","nonce":1291026539}]},"public_addr":"172.21.15.38:6817/1291026539","cluster_addr":"172.21.15.38:6819/1291026539","heartbeat_back_addr":"172.21.15.38:6823/1291026539","heartbeat_front_addr":"172.21.15.38:6821/1291026539","state":["exists","up"]},{"osd":1,"uuid":"7a899e9e-c234-4128-8b3c-5234e7a66aed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":2141962341},{"type":"v1","addr":"172.21.15.38:6809","nonce":2141962341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":2141962341},{"type":"v1","addr":"172.21.15.38:6811","nonce":2141962341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":2141962341},{"type":"v1","addr":"172.21.15.38:6815","nonce":2141962341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2141962341},{"type":"v1","addr":"172.21.15.38:6813","nonce":2141962341}]},"public_addr":"172.21.15.38:6809/2141962341","cluster_addr":"172.21.15.38:6811/2141962341","heartbeat_back_addr":"172.21.15.38:6815/2141962341","heartbeat_front_addr":"172.21.15.38:6813/2141962341","state":["exists","up"]},{"osd":2,"uuid":"06a70fc4-b466-44d3-9800-2e0c9780993e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":2430931791},{"type":"v1","addr":"172.21.15.38:6801","nonce":2430931791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":2430931791},{"type":"v1","addr":"172.21.15.38:6803","nonce":2430931791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2430931791},{"type":"v1","addr":"172.21.15.38:6807","nonce":2430931791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":2430931791},{"type":"v1","addr":"172.21.15.38:6805","nonce":2430931791}]},"public_addr":"172.21.15.38:6801/2430931791","cluster_addr":"172.21.15.38:6803/2430931791","heartbeat_back_addr":"172.21.15.38:6807/2430931791","heartbeat_front_addr":"172.21.15.38:6805/2430931791","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T23:05:04.623873+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T23:05:04.361551+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T23:05:04.333856+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T23:05:16.588 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.0 flush_pg_stats 2024-09-17T23:05:16.588 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.1 flush_pg_stats 2024-09-17T23:05:16.589 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.2 flush_pg_stats 2024-09-17T23:05:16.732 INFO:teuthology.orchestra.run.smithi038.stdout:38654705670 2024-09-17T23:05:16.732 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2024-09-17T23:05:16.733 INFO:teuthology.orchestra.run.smithi038.stdout:38654705669 2024-09-17T23:05:16.733 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2024-09-17T23:05:16.734 INFO:teuthology.orchestra.run.smithi038.stdout:38654705670 2024-09-17T23:05:16.734 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2024-09-17T23:05:17.069 INFO:teuthology.orchestra.run.smithi038.stdout:38654705669 2024-09-17T23:05:17.073 INFO:teuthology.orchestra.run.smithi038.stdout:38654705669 2024-09-17T23:05:17.077 INFO:teuthology.orchestra.run.smithi038.stdout:38654705667 2024-09-17T23:05:17.085 INFO:tasks.ceph.ceph_manager.cluster2:need seq 38654705670 got 38654705669 for osd.1 2024-09-17T23:05:17.088 INFO:tasks.ceph.ceph_manager.cluster2:need seq 38654705670 got 38654705669 for osd.2 2024-09-17T23:05:17.091 INFO:tasks.ceph.ceph_manager.cluster2:need seq 38654705669 got 38654705667 for osd.0 2024-09-17T23:05:18.086 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2024-09-17T23:05:18.089 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2024-09-17T23:05:18.091 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2024-09-17T23:05:18.416 INFO:teuthology.orchestra.run.smithi038.stdout:38654705669 2024-09-17T23:05:18.418 INFO:teuthology.orchestra.run.smithi038.stdout:38654705669 2024-09-17T23:05:18.430 INFO:tasks.ceph.ceph_manager.cluster2:need seq 38654705670 got 38654705669 for osd.1 2024-09-17T23:05:18.435 INFO:tasks.ceph.ceph_manager.cluster2:need seq 38654705670 got 38654705669 for osd.2 2024-09-17T23:05:18.440 INFO:teuthology.orchestra.run.smithi038.stdout:38654705667 2024-09-17T23:05:18.452 INFO:tasks.ceph.ceph_manager.cluster2:need seq 38654705669 got 38654705667 for osd.0 2024-09-17T23:05:19.430 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2024-09-17T23:05:19.437 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2024-09-17T23:05:19.453 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2024-09-17T23:05:19.761 INFO:teuthology.orchestra.run.smithi038.stdout:38654705671 2024-09-17T23:05:19.774 INFO:tasks.ceph.ceph_manager.cluster2:need seq 38654705670 got 38654705671 for osd.1 2024-09-17T23:05:19.774 DEBUG:teuthology.parallel:result is None 2024-09-17T23:05:19.778 INFO:teuthology.orchestra.run.smithi038.stdout:38654705671 2024-09-17T23:05:19.791 INFO:tasks.ceph.ceph_manager.cluster2:need seq 38654705670 got 38654705671 for osd.2 2024-09-17T23:05:19.791 DEBUG:teuthology.parallel:result is None 2024-09-17T23:05:19.802 INFO:teuthology.orchestra.run.smithi038.stdout:38654705669 2024-09-17T23:05:19.815 INFO:tasks.ceph.ceph_manager.cluster2:need seq 38654705669 got 38654705669 for osd.0 2024-09-17T23:05:19.815 DEBUG:teuthology.parallel:result is None 2024-09-17T23:05:19.815 INFO:tasks.ceph.ceph_manager.cluster2:waiting for clean 2024-09-17T23:05:19.815 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-17T23:05:20.145 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:05:20.146 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2024-09-17T23:05:20.157 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-09-17T23:05:18.508608+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":94120,"kb_used_data":552,"kb_used_omap":4,"kb_used_meta":93499,"kb_avail":283021400,"statfs":{"total":289910292480,"available":289813913600,"internally_reserved":0,"allocated":565248,"data_stored":1309576,"data_compressed":5998,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4785,"internal_metadata":95743311},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.913243"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-17T23:05:15.618717+0000","last_change":"2024-09-17T23:05:15.618717+0000","last_active":"2024-09-17T23:05:15.618717+0000","last_peered":"2024-09-17T23:05:15.618717+0000","last_clean":"2024-09-17T23:05:15.618717+0000","last_became_active":"2024-09-17T23:05:13.614964+0000","last_became_peered":"2024-09-17T23:05:13.614964+0000","last_unstale":"2024-09-17T23:05:15.618717+0000","last_undegraded":"2024-09-17T23:05:15.618717+0000","last_fullsized":"2024-09-17T23:05:15.618717+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:05:12.587919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:05:12.587919+0000","last_clean_scrub_stamp":"2024-09-17T23:05:12.587919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T02:02:11.663659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001157548,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-17T23:05:15.618037+0000","last_change":"2024-09-17T23:05:15.618037+0000","last_active":"2024-09-17T23:05:15.618037+0000","last_peered":"2024-09-17T23:05:15.618037+0000","last_clean":"2024-09-17T23:05:15.618037+0000","last_became_active":"2024-09-17T23:05:13.615179+0000","last_became_peered":"2024-09-17T23:05:13.615179+0000","last_unstale":"2024-09-17T23:05:15.618037+0000","last_undegraded":"2024-09-17T23:05:15.618037+0000","last_fullsized":"2024-09-17T23:05:15.618037+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:05:12.587919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:05:12.587919+0000","last_clean_scrub_stamp":"2024-09-17T23:05:12.587919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T07:27:12.449968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065459800000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-17T23:05:15.617997+0000","last_change":"2024-09-17T23:05:15.617997+0000","last_active":"2024-09-17T23:05:15.617997+0000","last_peered":"2024-09-17T23:05:15.617997+0000","last_clean":"2024-09-17T23:05:15.617997+0000","last_became_active":"2024-09-17T23:05:13.615245+0000","last_became_peered":"2024-09-17T23:05:13.615245+0000","last_unstale":"2024-09-17T23:05:15.617997+0000","last_undegraded":"2024-09-17T23:05:15.617997+0000","last_fullsized":"2024-09-17T23:05:15.617997+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:05:12.587919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:05:12.587919+0000","last_clean_scrub_stamp":"2024-09-17T23:05:12.587919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T05:02:26.092734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063722900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-17T23:05:15.618350+0000","last_change":"2024-09-17T23:05:15.618350+0000","last_active":"2024-09-17T23:05:15.618350+0000","last_peered":"2024-09-17T23:05:15.618350+0000","last_clean":"2024-09-17T23:05:15.618350+0000","last_became_active":"2024-09-17T23:05:13.615118+0000","last_became_peered":"2024-09-17T23:05:13.615118+0000","last_unstale":"2024-09-17T23:05:15.618350+0000","last_undegraded":"2024-09-17T23:05:15.618350+0000","last_fullsized":"2024-09-17T23:05:15.618350+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:05:12.587919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:05:12.587919+0000","last_clean_scrub_stamp":"2024-09-17T23:05:12.587919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T08:01:08.978654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097848599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"17'2","reported_seq":21,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-17T23:05:15.616751+0000","last_change":"2024-09-17T23:05:15.612022+0000","last_active":"2024-09-17T23:05:15.616751+0000","last_peered":"2024-09-17T23:05:15.616751+0000","last_clean":"2024-09-17T23:05:15.616751+0000","last_became_active":"2024-09-17T23:05:13.615350+0000","last_became_peered":"2024-09-17T23:05:13.615350+0000","last_unstale":"2024-09-17T23:05:15.616751+0000","last_undegraded":"2024-09-17T23:05:15.616751+0000","last_fullsized":"2024-09-17T23:05:15.616751+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:05:12.587919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:05:12.587919+0000","last_clean_scrub_stamp":"2024-09-17T23:05:12.587919+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T00:22:33.153175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049918900000000003,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-17T23:05:16.398203+0000","last_change":"2024-09-17T23:05:16.398203+0000","last_active":"2024-09-17T23:05:16.398203+0000","last_peered":"2024-09-17T23:05:16.398203+0000","last_clean":"2024-09-17T23:05:16.398203+0000","last_became_active":"2024-09-17T23:05:13.615287+0000","last_became_peered":"2024-09-17T23:05:13.615287+0000","last_unstale":"2024-09-17T23:05:16.398203+0000","last_undegraded":"2024-09-17T23:05:16.398203+0000","last_fullsized":"2024-09-17T23:05:16.398203+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:05:12.587919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:05:12.587919+0000","last_clean_scrub_stamp":"2024-09-17T23:05:12.587919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T10:27:38.318149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031014299999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-17T23:05:16.398209+0000","last_change":"2024-09-17T23:05:16.398209+0000","last_active":"2024-09-17T23:05:16.398209+0000","last_peered":"2024-09-17T23:05:16.398209+0000","last_clean":"2024-09-17T23:05:16.398209+0000","last_became_active":"2024-09-17T23:05:13.615033+0000","last_became_peered":"2024-09-17T23:05:13.615033+0000","last_unstale":"2024-09-17T23:05:16.398209+0000","last_undegraded":"2024-09-17T23:05:16.398209+0000","last_fullsized":"2024-09-17T23:05:16.398209+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:05:12.587919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:05:12.587919+0000","last_clean_scrub_stamp":"2024-09-17T23:05:12.587919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T01:58:05.905647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031522499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"15'1","reported_seq":20,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-17T23:05:15.618439+0000","last_change":"2024-09-17T23:05:15.618439+0000","last_active":"2024-09-17T23:05:15.618439+0000","last_peered":"2024-09-17T23:05:15.618439+0000","last_clean":"2024-09-17T23:05:15.618439+0000","last_became_active":"2024-09-17T23:05:13.609294+0000","last_became_peered":"2024-09-17T23:05:13.609294+0000","last_unstale":"2024-09-17T23:05:15.618439+0000","last_undegraded":"2024-09-17T23:05:15.618439+0000","last_fullsized":"2024-09-17T23:05:15.618439+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:05:12.587919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:05:12.587919+0000","last_clean_scrub_stamp":"2024-09-17T23:05:12.587919+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T02:57:25.565118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092743800000000005,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"12'97","reported_seq":147,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-17T23:05:15.617302+0000","last_change":"2024-09-17T23:05:09.702570+0000","last_active":"2024-09-17T23:05:15.617302+0000","last_peered":"2024-09-17T23:05:15.617302+0000","last_clean":"2024-09-17T23:05:15.617302+0000","last_became_active":"2024-09-17T23:05:09.701419+0000","last_became_peered":"2024-09-17T23:05:09.701419+0000","last_unstale":"2024-09-17T23:05:15.617302+0000","last_undegraded":"2024-09-17T23:05:15.617302+0000","last_fullsized":"2024-09-17T23:05:15.617302+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:05:08.557542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:05:08.557542+0000","last_clean_scrub_stamp":"2024-09-17T23:05:08.557542+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T08:10:48.938230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":5998,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705671,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31336,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340504,"statfs":{"total":96636764160,"available":96604676096,"internally_reserved":0,"allocated":147456,"data_stored":42934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705671,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31388,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340452,"statfs":{"total":96636764160,"available":96604622848,"internally_reserved":0,"allocated":208896,"data_stored":633321,"data_compressed":2999,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705669,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31396,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340444,"statfs":{"total":96636764160,"available":96604614656,"internally_reserved":0,"allocated":208896,"data_stored":633321,"data_compressed":2999,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2999,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2999,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T23:05:20.159 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-17T23:05:20.445 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:05:20.445 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2024-09-17T23:05:20.457 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-09-17T23:05:18.508608+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":94120,"kb_used_data":552,"kb_used_omap":4,"kb_used_meta":93499,"kb_avail":283021400,"statfs":{"total":289910292480,"available":289813913600,"internally_reserved":0,"allocated":565248,"data_stored":1309576,"data_compressed":5998,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4785,"internal_metadata":95743311},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.913243"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-17T23:05:15.618717+0000","last_change":"2024-09-17T23:05:15.618717+0000","last_active":"2024-09-17T23:05:15.618717+0000","last_peered":"2024-09-17T23:05:15.618717+0000","last_clean":"2024-09-17T23:05:15.618717+0000","last_became_active":"2024-09-17T23:05:13.614964+0000","last_became_peered":"2024-09-17T23:05:13.614964+0000","last_unstale":"2024-09-17T23:05:15.618717+0000","last_undegraded":"2024-09-17T23:05:15.618717+0000","last_fullsized":"2024-09-17T23:05:15.618717+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:05:12.587919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:05:12.587919+0000","last_clean_scrub_stamp":"2024-09-17T23:05:12.587919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T02:02:11.663659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001157548,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-17T23:05:15.618037+0000","last_change":"2024-09-17T23:05:15.618037+0000","last_active":"2024-09-17T23:05:15.618037+0000","last_peered":"2024-09-17T23:05:15.618037+0000","last_clean":"2024-09-17T23:05:15.618037+0000","last_became_active":"2024-09-17T23:05:13.615179+0000","last_became_peered":"2024-09-17T23:05:13.615179+0000","last_unstale":"2024-09-17T23:05:15.618037+0000","last_undegraded":"2024-09-17T23:05:15.618037+0000","last_fullsized":"2024-09-17T23:05:15.618037+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:05:12.587919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:05:12.587919+0000","last_clean_scrub_stamp":"2024-09-17T23:05:12.587919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T07:27:12.449968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065459800000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-17T23:05:15.617997+0000","last_change":"2024-09-17T23:05:15.617997+0000","last_active":"2024-09-17T23:05:15.617997+0000","last_peered":"2024-09-17T23:05:15.617997+0000","last_clean":"2024-09-17T23:05:15.617997+0000","last_became_active":"2024-09-17T23:05:13.615245+0000","last_became_peered":"2024-09-17T23:05:13.615245+0000","last_unstale":"2024-09-17T23:05:15.617997+0000","last_undegraded":"2024-09-17T23:05:15.617997+0000","last_fullsized":"2024-09-17T23:05:15.617997+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:05:12.587919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:05:12.587919+0000","last_clean_scrub_stamp":"2024-09-17T23:05:12.587919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T05:02:26.092734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063722900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-17T23:05:15.618350+0000","last_change":"2024-09-17T23:05:15.618350+0000","last_active":"2024-09-17T23:05:15.618350+0000","last_peered":"2024-09-17T23:05:15.618350+0000","last_clean":"2024-09-17T23:05:15.618350+0000","last_became_active":"2024-09-17T23:05:13.615118+0000","last_became_peered":"2024-09-17T23:05:13.615118+0000","last_unstale":"2024-09-17T23:05:15.618350+0000","last_undegraded":"2024-09-17T23:05:15.618350+0000","last_fullsized":"2024-09-17T23:05:15.618350+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:05:12.587919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:05:12.587919+0000","last_clean_scrub_stamp":"2024-09-17T23:05:12.587919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T08:01:08.978654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097848599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"17'2","reported_seq":21,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-17T23:05:15.616751+0000","last_change":"2024-09-17T23:05:15.612022+0000","last_active":"2024-09-17T23:05:15.616751+0000","last_peered":"2024-09-17T23:05:15.616751+0000","last_clean":"2024-09-17T23:05:15.616751+0000","last_became_active":"2024-09-17T23:05:13.615350+0000","last_became_peered":"2024-09-17T23:05:13.615350+0000","last_unstale":"2024-09-17T23:05:15.616751+0000","last_undegraded":"2024-09-17T23:05:15.616751+0000","last_fullsized":"2024-09-17T23:05:15.616751+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:05:12.587919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:05:12.587919+0000","last_clean_scrub_stamp":"2024-09-17T23:05:12.587919+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T00:22:33.153175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049918900000000003,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-17T23:05:16.398203+0000","last_change":"2024-09-17T23:05:16.398203+0000","last_active":"2024-09-17T23:05:16.398203+0000","last_peered":"2024-09-17T23:05:16.398203+0000","last_clean":"2024-09-17T23:05:16.398203+0000","last_became_active":"2024-09-17T23:05:13.615287+0000","last_became_peered":"2024-09-17T23:05:13.615287+0000","last_unstale":"2024-09-17T23:05:16.398203+0000","last_undegraded":"2024-09-17T23:05:16.398203+0000","last_fullsized":"2024-09-17T23:05:16.398203+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:05:12.587919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:05:12.587919+0000","last_clean_scrub_stamp":"2024-09-17T23:05:12.587919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T10:27:38.318149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031014299999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-17T23:05:16.398209+0000","last_change":"2024-09-17T23:05:16.398209+0000","last_active":"2024-09-17T23:05:16.398209+0000","last_peered":"2024-09-17T23:05:16.398209+0000","last_clean":"2024-09-17T23:05:16.398209+0000","last_became_active":"2024-09-17T23:05:13.615033+0000","last_became_peered":"2024-09-17T23:05:13.615033+0000","last_unstale":"2024-09-17T23:05:16.398209+0000","last_undegraded":"2024-09-17T23:05:16.398209+0000","last_fullsized":"2024-09-17T23:05:16.398209+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:05:12.587919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:05:12.587919+0000","last_clean_scrub_stamp":"2024-09-17T23:05:12.587919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T01:58:05.905647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031522499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"15'1","reported_seq":20,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-17T23:05:15.618439+0000","last_change":"2024-09-17T23:05:15.618439+0000","last_active":"2024-09-17T23:05:15.618439+0000","last_peered":"2024-09-17T23:05:15.618439+0000","last_clean":"2024-09-17T23:05:15.618439+0000","last_became_active":"2024-09-17T23:05:13.609294+0000","last_became_peered":"2024-09-17T23:05:13.609294+0000","last_unstale":"2024-09-17T23:05:15.618439+0000","last_undegraded":"2024-09-17T23:05:15.618439+0000","last_fullsized":"2024-09-17T23:05:15.618439+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:05:12.587919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:05:12.587919+0000","last_clean_scrub_stamp":"2024-09-17T23:05:12.587919+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T02:57:25.565118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092743800000000005,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"12'97","reported_seq":147,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-17T23:05:15.617302+0000","last_change":"2024-09-17T23:05:09.702570+0000","last_active":"2024-09-17T23:05:15.617302+0000","last_peered":"2024-09-17T23:05:15.617302+0000","last_clean":"2024-09-17T23:05:15.617302+0000","last_became_active":"2024-09-17T23:05:09.701419+0000","last_became_peered":"2024-09-17T23:05:09.701419+0000","last_unstale":"2024-09-17T23:05:15.617302+0000","last_undegraded":"2024-09-17T23:05:15.617302+0000","last_fullsized":"2024-09-17T23:05:15.617302+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:05:08.557542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:05:08.557542+0000","last_clean_scrub_stamp":"2024-09-17T23:05:08.557542+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T08:10:48.938230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":5998,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705671,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31336,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340504,"statfs":{"total":96636764160,"available":96604676096,"internally_reserved":0,"allocated":147456,"data_stored":42934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705671,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31388,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340452,"statfs":{"total":96636764160,"available":96604622848,"internally_reserved":0,"allocated":208896,"data_stored":633321,"data_compressed":2999,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705669,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31396,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340444,"statfs":{"total":96636764160,"available":96604614656,"internally_reserved":0,"allocated":208896,"data_stored":633321,"data_compressed":2999,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2999,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2999,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T23:05:20.458 INFO:tasks.ceph.ceph_manager.cluster2:clean! 2024-09-17T23:05:20.458 INFO:tasks.ceph:Waiting until ceph cluster cluster2 is healthy... 2024-09-17T23:05:20.459 INFO:tasks.ceph.ceph_manager.cluster2:wait_until_healthy 2024-09-17T23:05:20.459 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 health --format=json 2024-09-17T23:05:20.784 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:05:20.784 INFO:teuthology.orchestra.run.smithi038.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-17T23:05:20.796 INFO:tasks.ceph.ceph_manager.cluster2:wait_until_healthy done 2024-09-17T23:05:20.796 INFO:teuthology.run_tasks:Running task exec... 2024-09-17T23:05:20.811 INFO:teuthology.task.exec:Executing custom commands... 2024-09-17T23:05:20.811 INFO:teuthology.task.exec:Running commands on role cluster1.client.mirror.0 host ubuntu@smithi038.front.sepia.ceph.com 2024-09-17T23:05:20.811 DEBUG:teuthology.orchestra.run.smithi038:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror mon '"'"'profile rbd-mirror-peer'"'"' osd '"'"'profile rbd'"'"'' 2024-09-17T23:05:21.187 INFO:teuthology.orchestra.run.smithi038.stdout:[client.mirror] 2024-09-17T23:05:21.187 INFO:teuthology.orchestra.run.smithi038.stdout: key = AQDgCupmMGrABxAA0J818ImLp/GgA5cQRJYz2g== 2024-09-17T23:05:21.187 INFO:teuthology.orchestra.run.smithi038.stdout: caps mon = "profile rbd-mirror-peer" 2024-09-17T23:05:21.187 INFO:teuthology.orchestra.run.smithi038.stdout: caps osd = "profile rbd" 2024-09-17T23:05:21.187 INFO:teuthology.orchestra.run.smithi038.stderr:updated caps for client.mirror 2024-09-17T23:05:21.200 DEBUG:teuthology.orchestra.run.smithi038:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.0 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-17T23:05:21.569 INFO:teuthology.orchestra.run.smithi038.stdout:[client.mirror.0] 2024-09-17T23:05:21.570 INFO:teuthology.orchestra.run.smithi038.stdout: key = AQDgCupmWjAgCxAAefibdjhobV0WoLTqfqk4lQ== 2024-09-17T23:05:21.570 INFO:teuthology.orchestra.run.smithi038.stdout: caps mon = "profile rbd-mirror" 2024-09-17T23:05:21.570 INFO:teuthology.orchestra.run.smithi038.stdout: caps osd = "profile rbd" 2024-09-17T23:05:21.570 INFO:teuthology.orchestra.run.smithi038.stderr:updated caps for client.mirror.0 2024-09-17T23:05:21.583 DEBUG:teuthology.orchestra.run.smithi038:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.1 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-17T23:05:21.953 INFO:teuthology.orchestra.run.smithi038.stdout:[client.mirror.1] 2024-09-17T23:05:21.953 INFO:teuthology.orchestra.run.smithi038.stdout: key = AQDgCupm7h7VEBAArkVJ99C+P3IZ05GL9e/Fdw== 2024-09-17T23:05:21.953 INFO:teuthology.orchestra.run.smithi038.stdout: caps mon = "profile rbd-mirror" 2024-09-17T23:05:21.953 INFO:teuthology.orchestra.run.smithi038.stdout: caps osd = "profile rbd" 2024-09-17T23:05:21.953 INFO:teuthology.orchestra.run.smithi038.stderr:updated caps for client.mirror.1 2024-09-17T23:05:21.966 DEBUG:teuthology.orchestra.run.smithi038:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.2 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-17T23:05:22.333 INFO:teuthology.orchestra.run.smithi038.stdout:[client.mirror.2] 2024-09-17T23:05:22.334 INFO:teuthology.orchestra.run.smithi038.stdout: key = AQDgCupmobB6FhAA7cvicbHshpJh/r3Io3aUKw== 2024-09-17T23:05:22.334 INFO:teuthology.orchestra.run.smithi038.stdout: caps mon = "profile rbd-mirror" 2024-09-17T23:05:22.334 INFO:teuthology.orchestra.run.smithi038.stdout: caps osd = "profile rbd" 2024-09-17T23:05:22.334 INFO:teuthology.orchestra.run.smithi038.stderr:updated caps for client.mirror.2 2024-09-17T23:05:22.347 DEBUG:teuthology.orchestra.run.smithi038:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.3 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-17T23:05:22.717 INFO:teuthology.orchestra.run.smithi038.stdout:[client.mirror.3] 2024-09-17T23:05:22.717 INFO:teuthology.orchestra.run.smithi038.stdout: key = AQDgCupmqFYqHBAAd8snKc43AiQ6clVSoQSsPw== 2024-09-17T23:05:22.717 INFO:teuthology.orchestra.run.smithi038.stdout: caps mon = "profile rbd-mirror" 2024-09-17T23:05:22.718 INFO:teuthology.orchestra.run.smithi038.stdout: caps osd = "profile rbd" 2024-09-17T23:05:22.718 INFO:teuthology.orchestra.run.smithi038.stderr:updated caps for client.mirror.3 2024-09-17T23:05:22.732 DEBUG:teuthology.orchestra.run.smithi038:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror mon '"'"'profile rbd-mirror-peer'"'"' osd '"'"'profile rbd'"'"'' 2024-09-17T23:05:23.106 INFO:teuthology.orchestra.run.smithi038.stdout:[client.mirror] 2024-09-17T23:05:23.106 INFO:teuthology.orchestra.run.smithi038.stdout: key = AQAHC+pmTdhiGxAAft2fvH1QDiXNxRJwcJLNpw== 2024-09-17T23:05:23.106 INFO:teuthology.orchestra.run.smithi038.stdout: caps mon = "profile rbd-mirror-peer" 2024-09-17T23:05:23.106 INFO:teuthology.orchestra.run.smithi038.stdout: caps osd = "profile rbd" 2024-09-17T23:05:23.107 INFO:teuthology.orchestra.run.smithi038.stderr:updated caps for client.mirror 2024-09-17T23:05:23.120 DEBUG:teuthology.orchestra.run.smithi038:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.0 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-17T23:05:23.494 INFO:teuthology.orchestra.run.smithi038.stdout:[client.mirror.0] 2024-09-17T23:05:23.494 INFO:teuthology.orchestra.run.smithi038.stdout: key = AQAHC+pmEzQ4IRAA0pSkPl5sElORKwl87e1NCQ== 2024-09-17T23:05:23.494 INFO:teuthology.orchestra.run.smithi038.stdout: caps mon = "profile rbd-mirror" 2024-09-17T23:05:23.494 INFO:teuthology.orchestra.run.smithi038.stdout: caps osd = "profile rbd" 2024-09-17T23:05:23.494 INFO:teuthology.orchestra.run.smithi038.stderr:updated caps for client.mirror.0 2024-09-17T23:05:23.507 DEBUG:teuthology.orchestra.run.smithi038:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.1 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-17T23:05:23.875 INFO:teuthology.orchestra.run.smithi038.stdout:[client.mirror.1] 2024-09-17T23:05:23.875 INFO:teuthology.orchestra.run.smithi038.stdout: key = AQAHC+pmdRaAJxAAnYqUAXNFzNwm/Ni/eIMMFw== 2024-09-17T23:05:23.875 INFO:teuthology.orchestra.run.smithi038.stdout: caps mon = "profile rbd-mirror" 2024-09-17T23:05:23.875 INFO:teuthology.orchestra.run.smithi038.stdout: caps osd = "profile rbd" 2024-09-17T23:05:23.875 INFO:teuthology.orchestra.run.smithi038.stderr:updated caps for client.mirror.1 2024-09-17T23:05:23.889 DEBUG:teuthology.orchestra.run.smithi038:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.2 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-17T23:05:24.254 INFO:teuthology.orchestra.run.smithi038.stdout:[client.mirror.2] 2024-09-17T23:05:24.254 INFO:teuthology.orchestra.run.smithi038.stdout: key = AQAHC+pmUUU4LRAAJQ97LEyaJosHdgSb5DXn4Q== 2024-09-17T23:05:24.254 INFO:teuthology.orchestra.run.smithi038.stdout: caps mon = "profile rbd-mirror" 2024-09-17T23:05:24.254 INFO:teuthology.orchestra.run.smithi038.stdout: caps osd = "profile rbd" 2024-09-17T23:05:24.255 INFO:teuthology.orchestra.run.smithi038.stderr:updated caps for client.mirror.2 2024-09-17T23:05:24.268 DEBUG:teuthology.orchestra.run.smithi038:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.3 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-17T23:05:24.635 INFO:teuthology.orchestra.run.smithi038.stdout:[client.mirror.3] 2024-09-17T23:05:24.635 INFO:teuthology.orchestra.run.smithi038.stdout: key = AQAHC+pmFiXxMhAA207RAyAmrhRt5+u0Nkv/rw== 2024-09-17T23:05:24.635 INFO:teuthology.orchestra.run.smithi038.stdout: caps mon = "profile rbd-mirror" 2024-09-17T23:05:24.635 INFO:teuthology.orchestra.run.smithi038.stdout: caps osd = "profile rbd" 2024-09-17T23:05:24.635 INFO:teuthology.orchestra.run.smithi038.stderr:updated caps for client.mirror.3 2024-09-17T23:05:24.648 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-09-17T23:05:24.660 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-09-17T23:05:24.661 DEBUG:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-09-17T23:05:24.663 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-09-17T23:05:24.663 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-09-17T23:05:24.671 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2024-09-17T23:05:24.671 DEBUG:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2024-09-17T23:05:24.673 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2024-09-17T23:05:24.673 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-09-17T23:05:24.681 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2024-09-17T23:05:24.681 DEBUG:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2024-09-17T23:05:24.683 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2024-09-17T23:05:24.683 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-09-17T23:05:24.691 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2024-09-17T23:05:24.692 DEBUG:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2024-09-17T23:05:24.694 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2024-09-17T23:05:24.694 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-09-17T23:05:24.702 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-09-17T23:05:24.702 DEBUG:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-09-17T23:05:24.705 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-09-17T23:05:24.705 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-09-17T23:05:24.713 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2024-09-17T23:05:24.713 DEBUG:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2024-09-17T23:05:24.716 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2024-09-17T23:05:24.716 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-09-17T23:05:24.724 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2024-09-17T23:05:24.724 DEBUG:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2024-09-17T23:05:24.726 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2024-09-17T23:05:24.726 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-09-17T23:05:24.735 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2024-09-17T23:05:24.735 DEBUG:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2024-09-17T23:05:24.739 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2024-09-17T23:05:24.740 INFO:teuthology.run_tasks:Running task rbd-mirror-thrash... 2024-09-17T23:05:24.751 INFO:tasks.rbd_mirror_thrash:rbd_mirror_thrash using random seed: 1726614324 2024-09-17T23:05:24.751 DEBUG:tasks.rbd_mirror_thrash:Yielding 2024-09-17T23:05:24.751 INFO:teuthology.run_tasks:Running task rbd-mirror-thrash... 2024-09-17T23:05:24.760 INFO:tasks.rbd_mirror_thrash:rbd_mirror_thrash using random seed: 1726614324 2024-09-17T23:05:24.760 DEBUG:tasks.rbd_mirror_thrash:Yielding 2024-09-17T23:05:24.760 INFO:teuthology.run_tasks:Running task workunit... 2024-09-17T23:05:24.774 INFO:tasks.workunit:Pulling workunits from ref b0d356b98e08f6bc59ef15652f0d0e4298fc83c7 2024-09-17T23:05:24.774 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-17T23:05:24.774 DEBUG:teuthology.orchestra.run.smithi038:> stat -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-09-17T23:05:24.775 INFO:tasks.rbd_mirror_thrash:starting thrash for cluster cluster1 2024-09-17T23:05:24.775 INFO:tasks.rbd_mirror_thrash:waiting for 66 secs before thrashing 2024-09-17T23:05:24.775 INFO:tasks.rbd_mirror_thrash:starting thrash for cluster cluster2 2024-09-17T23:05:24.775 INFO:tasks.rbd_mirror_thrash:waiting for 113 secs before thrashing 2024-09-17T23:05:24.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T23:05:24.798 INFO:teuthology.orchestra.run.smithi038.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.cluster1.mirror': No such file or directory 2024-09-17T23:05:24.798 DEBUG:teuthology.orchestra.run.smithi038:> mkdir -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-09-17T23:05:24.854 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-09-17T23:05:24.854 DEBUG:teuthology.orchestra.run.smithi038:> cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror && mkdir -- client.mirror 2024-09-17T23:05:24.910 INFO:tasks.workunit:timeout=6h 2024-09-17T23:05:24.910 INFO:tasks.workunit:cleanup=True 2024-09-17T23:05:24.910 DEBUG:teuthology.orchestra.run.smithi038:> rm -rf /home/ubuntu/cephtest/clone.cluster1.client.mirror && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.cluster1.client.mirror && cd /home/ubuntu/cephtest/clone.cluster1.client.mirror && git checkout b0d356b98e08f6bc59ef15652f0d0e4298fc83c7 2024-09-17T23:05:24.967 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:Cloning into '/home/ubuntu/cephtest/clone.cluster1.client.mirror'... 2024-09-17T23:06:21.129 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:Updating files: 93% (12048/12893) Updating files: 94% (12120/12893) Updating files: 95% (12249/12893) Updating files: 96% (12378/12893) Updating files: 97% (12507/12893) Updating files: 98% (12636/12893) Updating files: 99% (12765/12893) Updating files: 100% (12893/12893) Updating files: 100% (12893/12893), done. 2024-09-17T23:06:21.563 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:Note: switching to 'b0d356b98e08f6bc59ef15652f0d0e4298fc83c7'. 2024-09-17T23:06:21.563 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:06:21.563 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-17T23:06:21.563 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-17T23:06:21.563 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:state without impacting any branches by switching back to a branch. 2024-09-17T23:06:21.563 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:06:21.563 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-17T23:06:21.563 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-17T23:06:21.563 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:06:21.563 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: git switch -c 2024-09-17T23:06:21.563 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:06:21.564 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:Or undo this operation with: 2024-09-17T23:06:21.564 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:06:21.564 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: git switch - 2024-09-17T23:06:21.564 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:06:21.564 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-17T23:06:21.564 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:06:21.564 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:HEAD is now at b0d356b98e0 Merge pull request #59811 from afreen23/wip-68079-squid 2024-09-17T23:06:21.567 DEBUG:teuthology.orchestra.run.smithi038:> cd -- /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.cluster1.client.mirror 2024-09-17T23:06:21.635 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-09-17T23:06:21.637 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/direct_io' 2024-09-17T23:06:21.638 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-09-17T23:06:22.333 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-09-17T23:06:22.404 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-09-17T23:06:22.445 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/direct_io' 2024-09-17T23:06:22.447 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/fs' 2024-09-17T23:06:22.447 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-09-17T23:06:22.487 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/fs' 2024-09-17T23:06:22.490 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-09-17T23:06:22.490 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/home/ubuntu/cephtest/workunits.list.cluster1.client.mirror of=/dev/stdout 2024-09-17T23:06:22.546 INFO:tasks.workunit:Running workunits matching rbd/rbd_mirror_stress.sh on cluster1.client.mirror... 2024-09-17T23:06:22.547 INFO:tasks.workunit:Running workunit rbd/rbd_mirror_stress.sh... 2024-09-17T23:06:22.548 DEBUG:teuthology.orchestra.run.smithi038:workunit test rbd/rbd_mirror_stress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b0d356b98e08f6bc59ef15652f0d0e4298fc83c7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS='' MIRROR_POOL_MODE=image RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff RBD_MIRROR_INSTANCES=4 RBD_MIRROR_MODE=snapshot RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2024-09-17T23:06:22.607 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ IMAGE_COUNT=50 2024-09-17T23:06:22.607 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ export LOCKDEP=0 2024-09-17T23:06:22.607 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ LOCKDEP=0 2024-09-17T23:06:22.607 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ dirname /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2024-09-17T23:06:22.608 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ . /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_helpers.sh 2024-09-17T23:06:22.608 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ type xmlstarlet 2024-09-17T23:06:22.608 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ XMLSTARLET=xmlstarlet 2024-09-17T23:06:22.609 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ RBD_MIRROR_INSTANCES=4 2024-09-17T23:06:22.609 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ CLUSTER1=cluster1 2024-09-17T23:06:22.609 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ CLUSTER2=cluster2 2024-09-17T23:06:22.609 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ PEER_CLUSTER_SUFFIX= 2024-09-17T23:06:22.609 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ POOL=mirror 2024-09-17T23:06:22.609 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ PARENT_POOL=mirror_parent 2024-09-17T23:06:22.609 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ NS1=ns1 2024-09-17T23:06:22.609 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ NS2=ns2 2024-09-17T23:06:22.609 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ TEMPDIR= 2024-09-17T23:06:22.609 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ CEPH_ID=mirror 2024-09-17T23:06:22.609 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff 2024-09-17T23:06:22.609 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ MIRROR_USER_ID_PREFIX=mirror. 2024-09-17T23:06:22.609 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ RBD_MIRROR_MODE=snapshot 2024-09-17T23:06:22.610 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ MIRROR_POOL_MODE=image 2024-09-17T23:06:22.610 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' snapshot = snapshot ']' 2024-09-17T23:06:22.610 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ MIRROR_POOL_MODE=image 2024-09-17T23:06:22.610 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ export 'CEPH_ARGS=--id mirror' 2024-09-17T23:06:22.610 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ CEPH_ARGS='--id mirror' 2024-09-17T23:06:22.610 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ LAST_MIRROR_INSTANCE=3 2024-09-17T23:06:22.610 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ dirname /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2024-09-17T23:06:22.610 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ readlink -f /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/../../../src 2024-09-17T23:06:22.611 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror/src 2024-09-17T23:06:22.611 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ CEPH_BIN=. 2024-09-17T23:06:22.611 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ CEPH_SRC=. 2024-09-17T23:06:22.611 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' -e CMakeCache.txt ']' 2024-09-17T23:06:22.612 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 0 -gt 0 ']' 2024-09-17T23:06:22.612 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ setup 2024-09-17T23:06:22.612 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local c 2024-09-17T23:06:22.612 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ trap 'cleanup $?' INT TERM EXIT 2024-09-17T23:06:22.612 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ setup_tempdir 2024-09-17T23:06:22.613 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n '' ']' 2024-09-17T23:06:22.613 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mktemp -d 2024-09-17T23:06:22.613 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ TEMPDIR=/tmp/tmp.s5MEFDgsej 2024-09-17T23:06:22.613 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -z 1 ']' 2024-09-17T23:06:22.614 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ setup_pools cluster1 cluster2 2024-09-17T23:06:22.614 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:06:22.614 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local remote_cluster=cluster2 2024-09-17T23:06:22.614 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mon_map_file 2024-09-17T23:06:22.614 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mon_addr 2024-09-17T23:06:22.614 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local admin_key_file 2024-09-17T23:06:22.614 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local uuid 2024-09-17T23:06:22.614 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ CEPH_ARGS= 2024-09-17T23:06:22.614 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ ceph --cluster cluster1 osd pool create mirror 64 64 2024-09-17T23:06:23.339 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:pool 'mirror' already exists 2024-09-17T23:06:23.349 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ CEPH_ARGS= 2024-09-17T23:06:23.349 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ ceph --cluster cluster1 osd pool create mirror_parent 64 64 2024-09-17T23:06:24.346 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:pool 'mirror_parent' already exists 2024-09-17T23:06:24.357 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ CEPH_ARGS= 2024-09-17T23:06:24.357 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 pool init mirror 2024-09-17T23:06:27.263 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ CEPH_ARGS= 2024-09-17T23:06:27.264 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 pool init mirror_parent 2024-09-17T23:06:30.281 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n '' ']' 2024-09-17T23:06:30.282 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ CEPH_ARGS= 2024-09-17T23:06:30.282 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 mirror pool enable --site-name cluster1 mirror image 2024-09-17T23:06:30.336 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 mirror pool enable mirror_parent image 2024-09-17T23:06:30.379 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 namespace create mirror/ns1 2024-09-17T23:06:30.421 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 namespace create mirror/ns2 2024-09-17T23:06:30.458 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 mirror pool enable mirror/ns1 image 2024-09-17T23:06:30.499 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 mirror pool enable mirror/ns2 image 2024-09-17T23:06:30.542 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -z ']' 2024-09-17T23:06:30.542 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -z ']' 2024-09-17T23:06:30.542 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ peer_add cluster1 mirror cluster2 2024-09-17T23:06:30.542 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:06:30.542 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:06:30.542 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:06:30.542 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:06:30.542 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local client_cluster=cluster2 2024-09-17T23:06:30.542 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:06:30.542 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local remote_cluster=cluster2 2024-09-17T23:06:30.542 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local uuid_var_name 2024-09-17T23:06:30.543 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n '' ']' 2024-09-17T23:06:30.543 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local error_code 2024-09-17T23:06:30.543 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local peer_uuid 2024-09-17T23:06:30.543 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 16 32 2024-09-17T23:06:30.543 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set +e 2024-09-17T23:06:30.543 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster cluster1 mirror pool peer add mirror cluster2 2024-09-17T23:06:30.582 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ peer_uuid=803aac29-dea2-475c-a073-828f4b7740a3 2024-09-17T23:06:30.582 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ error_code=0 2024-09-17T23:06:30.582 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set -e 2024-09-17T23:06:30.582 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' 0 -eq 17 ']' 2024-09-17T23:06:30.582 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 0 -eq 0 2024-09-17T23:06:30.582 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n '' ']' 2024-09-17T23:06:30.582 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:06:30.582 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ peer_add cluster1 mirror_parent cluster2 2024-09-17T23:06:30.582 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:06:30.582 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:06:30.582 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror_parent 2024-09-17T23:06:30.582 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:06:30.582 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local client_cluster=cluster2 2024-09-17T23:06:30.583 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:06:30.583 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local remote_cluster=cluster2 2024-09-17T23:06:30.583 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local uuid_var_name 2024-09-17T23:06:30.583 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n '' ']' 2024-09-17T23:06:30.583 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local error_code 2024-09-17T23:06:30.583 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local peer_uuid 2024-09-17T23:06:30.583 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 16 32 2024-09-17T23:06:30.583 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set +e 2024-09-17T23:06:30.583 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster cluster1 mirror pool peer add mirror_parent cluster2 2024-09-17T23:06:30.621 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ peer_uuid=f5a8ac7c-03fd-4f9c-8233-cb887578c29b 2024-09-17T23:06:30.621 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ error_code=0 2024-09-17T23:06:30.621 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set -e 2024-09-17T23:06:30.621 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' 0 -eq 17 ']' 2024-09-17T23:06:30.622 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 0 -eq 0 2024-09-17T23:06:30.622 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n '' ']' 2024-09-17T23:06:30.622 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:06:30.622 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ setup_pools cluster2 cluster1 2024-09-17T23:06:30.622 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:06:30.622 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local remote_cluster=cluster1 2024-09-17T23:06:30.622 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mon_map_file 2024-09-17T23:06:30.622 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mon_addr 2024-09-17T23:06:30.622 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local admin_key_file 2024-09-17T23:06:30.622 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local uuid 2024-09-17T23:06:30.622 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ CEPH_ARGS= 2024-09-17T23:06:30.622 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ ceph --cluster cluster2 osd pool create mirror 64 64 2024-09-17T23:06:30.775 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.8393498161176968) > weight (0.25) 2024-09-17T23:06:30.776 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2024-09-17T23:06:30.776 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2024-09-17T23:06:30.776 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2024-09-17T23:06:30.777 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi038.stderr:2024-09-17T23:06:30.773+0000 7f5aae50f640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 36639) UID: 1000 2024-09-17T23:06:31.777 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2024-09-17T23:06:31.777 INFO:teuthology.orchestra.run:waiting for 600 2024-09-17T23:06:31.777 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2024-09-17T23:06:31.777 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2024-09-17T23:06:31.777 DEBUG:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2024-09-17T23:06:31.780 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2024-09-17T23:06:31.780 INFO:tasks.rbd_mirror_thrash:waiting for 61 secs before thrashing 2024-09-17T23:06:31.833 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:pool 'mirror' already exists 2024-09-17T23:06:31.845 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ CEPH_ARGS= 2024-09-17T23:06:31.845 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ ceph --cluster cluster2 osd pool create mirror_parent 64 64 2024-09-17T23:06:32.772 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi038.stderr:2024-09-17T23:06:32.769+0000 7f5a4ac0c640 -1 rbd::mirror::PoolReplayer: 0x55dc1f6c3b00 init: failed to initialize remote pool poller: (2) No such file or directory 2024-09-17T23:06:32.862 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:pool 'mirror_parent' already exists 2024-09-17T23:06:32.873 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ CEPH_ARGS= 2024-09-17T23:06:32.873 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 pool init mirror 2024-09-17T23:06:33.758 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi038.stderr:2024-09-17T23:06:33.756+0000 7f5a4ac0c640 -1 rbd::mirror::PoolReplayer: 0x55dc2077b800 init: failed to initialize remote pool poller: (2) No such file or directory 2024-09-17T23:06:35.746 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ CEPH_ARGS= 2024-09-17T23:06:35.747 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 pool init mirror_parent 2024-09-17T23:06:38.787 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n '' ']' 2024-09-17T23:06:38.787 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ CEPH_ARGS= 2024-09-17T23:06:38.788 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 mirror pool enable --site-name cluster2 mirror image 2024-09-17T23:06:38.838 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 mirror pool enable mirror_parent image 2024-09-17T23:06:38.879 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 namespace create mirror/ns1 2024-09-17T23:06:38.918 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 namespace create mirror/ns2 2024-09-17T23:06:38.955 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 mirror pool enable mirror/ns1 image 2024-09-17T23:06:38.993 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 mirror pool enable mirror/ns2 image 2024-09-17T23:06:39.033 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -z ']' 2024-09-17T23:06:39.034 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -z ']' 2024-09-17T23:06:39.034 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ peer_add cluster2 mirror cluster1 2024-09-17T23:06:39.034 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:06:39.034 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:06:39.034 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:06:39.034 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:06:39.034 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local client_cluster=cluster1 2024-09-17T23:06:39.034 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:06:39.034 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local remote_cluster=cluster1 2024-09-17T23:06:39.034 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local uuid_var_name 2024-09-17T23:06:39.034 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n '' ']' 2024-09-17T23:06:39.034 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local error_code 2024-09-17T23:06:39.035 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local peer_uuid 2024-09-17T23:06:39.035 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 16 32 2024-09-17T23:06:39.035 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set +e 2024-09-17T23:06:39.035 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster cluster2 mirror pool peer add mirror cluster1 2024-09-17T23:06:39.071 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ peer_uuid=0e8c665c-67c8-43be-a771-17ae8fdd5dc5 2024-09-17T23:06:39.072 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ error_code=0 2024-09-17T23:06:39.072 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set -e 2024-09-17T23:06:39.072 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' 0 -eq 17 ']' 2024-09-17T23:06:39.072 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 0 -eq 0 2024-09-17T23:06:39.072 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n '' ']' 2024-09-17T23:06:39.072 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:06:39.072 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ peer_add cluster2 mirror_parent cluster1 2024-09-17T23:06:39.072 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:06:39.072 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:06:39.072 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror_parent 2024-09-17T23:06:39.072 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:06:39.072 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local client_cluster=cluster1 2024-09-17T23:06:39.072 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:06:39.072 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local remote_cluster=cluster1 2024-09-17T23:06:39.073 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local uuid_var_name 2024-09-17T23:06:39.073 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n '' ']' 2024-09-17T23:06:39.073 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local error_code 2024-09-17T23:06:39.073 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local peer_uuid 2024-09-17T23:06:39.073 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 16 32 2024-09-17T23:06:39.073 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set +e 2024-09-17T23:06:39.073 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster cluster2 mirror pool peer add mirror_parent cluster1 2024-09-17T23:06:39.109 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ peer_uuid=7fcde847-128c-4311-ba71-be3574fa77a0 2024-09-17T23:06:39.109 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ error_code=0 2024-09-17T23:06:39.110 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set -e 2024-09-17T23:06:39.110 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' 0 -eq 17 ']' 2024-09-17T23:06:39.110 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 0 -eq 0 2024-09-17T23:06:39.110 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n '' ']' 2024-09-17T23:06:39.110 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:06:39.110 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n '' ']' 2024-09-17T23:06:39.110 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ start_mirrors cluster1 2024-09-17T23:06:39.110 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:06:39.110 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ seq 0 3 2024-09-17T23:06:39.140 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:06:39.140 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ start_mirror cluster1:0 2024-09-17T23:06:39.140 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1:0 2024-09-17T23:06:39.140 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local instance 2024-09-17T23:06:39.140 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-17T23:06:39.140 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local val=cluster1:0 2024-09-17T23:06:39.141 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster_var_name=cluster 2024-09-17T23:06:39.141 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local instance_var_name=instance 2024-09-17T23:06:39.141 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ cluster=cluster1 2024-09-17T23:06:39.141 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ instance=0 2024-09-17T23:06:39.141 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-17T23:06:39.141 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ eval cluster=cluster1 2024-09-17T23:06:39.141 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:06:39.141 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ eval instance=0 2024-09-17T23:06:39.141 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=0 2024-09-17T23:06:39.141 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n 1 2024-09-17T23:06:39.141 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 2024-09-17T23:06:39.141 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:06:39.141 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ start_mirror cluster1:1 2024-09-17T23:06:39.142 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1:1 2024-09-17T23:06:39.142 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local instance 2024-09-17T23:06:39.142 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-17T23:06:39.142 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local val=cluster1:1 2024-09-17T23:06:39.142 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster_var_name=cluster 2024-09-17T23:06:39.142 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local instance_var_name=instance 2024-09-17T23:06:39.142 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ cluster=cluster1 2024-09-17T23:06:39.142 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ instance=1 2024-09-17T23:06:39.142 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-17T23:06:39.142 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ eval cluster=cluster1 2024-09-17T23:06:39.142 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:06:39.142 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ eval instance=1 2024-09-17T23:06:39.142 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=1 2024-09-17T23:06:39.143 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n 1 2024-09-17T23:06:39.143 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 2024-09-17T23:06:39.143 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:06:39.143 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ start_mirror cluster1:2 2024-09-17T23:06:39.143 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1:2 2024-09-17T23:06:39.143 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local instance 2024-09-17T23:06:39.143 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-09-17T23:06:39.143 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local val=cluster1:2 2024-09-17T23:06:39.143 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster_var_name=cluster 2024-09-17T23:06:39.143 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local instance_var_name=instance 2024-09-17T23:06:39.143 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ cluster=cluster1 2024-09-17T23:06:39.143 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ instance=2 2024-09-17T23:06:39.143 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' 2 = cluster1:2 ']' 2024-09-17T23:06:39.143 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ eval cluster=cluster1 2024-09-17T23:06:39.144 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:06:39.144 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ eval instance=2 2024-09-17T23:06:39.144 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=2 2024-09-17T23:06:39.144 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n 1 2024-09-17T23:06:39.144 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 2024-09-17T23:06:39.144 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:06:39.144 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ start_mirror cluster1:3 2024-09-17T23:06:39.144 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1:3 2024-09-17T23:06:39.144 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local instance 2024-09-17T23:06:39.144 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-09-17T23:06:39.144 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local val=cluster1:3 2024-09-17T23:06:39.144 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster_var_name=cluster 2024-09-17T23:06:39.144 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local instance_var_name=instance 2024-09-17T23:06:39.144 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ cluster=cluster1 2024-09-17T23:06:39.145 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ instance=3 2024-09-17T23:06:39.145 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' 3 = cluster1:3 ']' 2024-09-17T23:06:39.145 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ eval cluster=cluster1 2024-09-17T23:06:39.145 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:06:39.145 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ eval instance=3 2024-09-17T23:06:39.145 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=3 2024-09-17T23:06:39.145 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n 1 2024-09-17T23:06:39.145 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 2024-09-17T23:06:39.145 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ start_mirrors cluster2 2024-09-17T23:06:39.145 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:06:39.145 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ seq 0 3 2024-09-17T23:06:39.145 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:06:39.145 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ start_mirror cluster2:0 2024-09-17T23:06:39.145 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2:0 2024-09-17T23:06:39.146 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local instance 2024-09-17T23:06:39.146 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-09-17T23:06:39.146 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local val=cluster2:0 2024-09-17T23:06:39.146 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster_var_name=cluster 2024-09-17T23:06:39.146 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local instance_var_name=instance 2024-09-17T23:06:39.146 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ cluster=cluster2 2024-09-17T23:06:39.146 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ instance=0 2024-09-17T23:06:39.146 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' 0 = cluster2:0 ']' 2024-09-17T23:06:39.146 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ eval cluster=cluster2 2024-09-17T23:06:39.146 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster2 2024-09-17T23:06:39.146 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ eval instance=0 2024-09-17T23:06:39.146 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=0 2024-09-17T23:06:39.146 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n 1 2024-09-17T23:06:39.146 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 2024-09-17T23:06:39.147 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:06:39.147 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ start_mirror cluster2:1 2024-09-17T23:06:39.147 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2:1 2024-09-17T23:06:39.147 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local instance 2024-09-17T23:06:39.147 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set_cluster_instance cluster2:1 cluster instance 2024-09-17T23:06:39.147 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local val=cluster2:1 2024-09-17T23:06:39.147 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster_var_name=cluster 2024-09-17T23:06:39.147 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local instance_var_name=instance 2024-09-17T23:06:39.147 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ cluster=cluster2 2024-09-17T23:06:39.147 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ instance=1 2024-09-17T23:06:39.147 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' 1 = cluster2:1 ']' 2024-09-17T23:06:39.147 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ eval cluster=cluster2 2024-09-17T23:06:39.147 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster2 2024-09-17T23:06:39.147 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ eval instance=1 2024-09-17T23:06:39.148 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=1 2024-09-17T23:06:39.148 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n 1 2024-09-17T23:06:39.148 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 2024-09-17T23:06:39.148 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:06:39.148 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ start_mirror cluster2:2 2024-09-17T23:06:39.148 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2:2 2024-09-17T23:06:39.148 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local instance 2024-09-17T23:06:39.148 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set_cluster_instance cluster2:2 cluster instance 2024-09-17T23:06:39.148 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local val=cluster2:2 2024-09-17T23:06:39.148 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster_var_name=cluster 2024-09-17T23:06:39.148 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local instance_var_name=instance 2024-09-17T23:06:39.148 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ cluster=cluster2 2024-09-17T23:06:39.148 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ instance=2 2024-09-17T23:06:39.148 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' 2 = cluster2:2 ']' 2024-09-17T23:06:39.148 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ eval cluster=cluster2 2024-09-17T23:06:39.149 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster2 2024-09-17T23:06:39.149 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ eval instance=2 2024-09-17T23:06:39.149 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=2 2024-09-17T23:06:39.149 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n 1 2024-09-17T23:06:39.149 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 2024-09-17T23:06:39.149 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:06:39.149 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ start_mirror cluster2:3 2024-09-17T23:06:39.149 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2:3 2024-09-17T23:06:39.149 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local instance 2024-09-17T23:06:39.149 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set_cluster_instance cluster2:3 cluster instance 2024-09-17T23:06:39.149 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local val=cluster2:3 2024-09-17T23:06:39.149 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster_var_name=cluster 2024-09-17T23:06:39.149 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local instance_var_name=instance 2024-09-17T23:06:39.149 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ cluster=cluster2 2024-09-17T23:06:39.150 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ instance=3 2024-09-17T23:06:39.150 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' 3 = cluster2:3 ']' 2024-09-17T23:06:39.150 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ eval cluster=cluster2 2024-09-17T23:06:39.150 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster2 2024-09-17T23:06:39.150 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ eval instance=3 2024-09-17T23:06:39.150 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=3 2024-09-17T23:06:39.150 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n 1 2024-09-17T23:06:39.150 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 2024-09-17T23:06:39.150 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ testlog 'TEST: add image and test replay after client crashes' 2024-09-17T23:06:39.150 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ tee -a /tmp/tmp.s5MEFDgsej/rbd-mirror.test.log 2024-09-17T23:06:39.150 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ date '+%F %T' 2024-09-17T23:06:39.151 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ echo 2024-09-17 23:06:39 TEST: add image and test replay after client crashes 2024-09-17T23:06:39.151 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:2024-09-17 23:06:39 TEST: add image and test replay after client crashes 2024-09-17T23:06:39.151 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=test 2024-09-17T23:06:39.151 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_image_and_enable_mirror cluster2 mirror test snapshot 512M 2024-09-17T23:06:39.151 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:06:39.151 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:06:39.151 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:06:39.151 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:06:39.151 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:06:39.151 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:06:39.151 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=snapshot 2024-09-17T23:06:39.151 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n snapshot ']' 2024-09-17T23:06:39.151 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:06:39.151 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_image cluster2 mirror test 512M 2024-09-17T23:06:39.151 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:06:39.152 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:06:39.152 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:06:39.152 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:06:39.152 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:06:39.152 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:06:39.152 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local size=128 2024-09-17T23:06:39.152 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n 512M ']' 2024-09-17T23:06:39.152 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ size=512M 2024-09-17T23:06:39.152 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:06:39.152 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/test 2024-09-17T23:06:39.181 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ is_pool_mirror_mode_image mirror 2024-09-17T23:06:39.181 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:06:39.182 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' image = image ']' 2024-09-17T23:06:39.182 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:06:39.182 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ enable_mirror cluster2 mirror test snapshot 2024-09-17T23:06:39.182 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:06:39.182 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:06:39.182 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:06:39.182 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=snapshot 2024-09-17T23:06:39.182 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/test snapshot 2024-09-17T23:06:39.854 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Mirroring enabled 2024-09-17T23:06:39.863 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 info mirror/test 2024-09-17T23:06:39.903 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'test': 2024-09-17T23:06:39.903 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 512 MiB in 128 objects 2024-09-17T23:06:39.903 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:06:39.904 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 1 2024-09-17T23:06:39.904 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 10b37f53ed7e 2024-09-17T23:06:39.904 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.10b37f53ed7e 2024-09-17T23:06:39.904 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:06:39.904 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-17T23:06:39.904 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:06:39.904 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:06:39.904 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:06:39 2024 2024-09-17T23:06:39.904 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:06:39 2024 2024-09-17T23:06:39.904 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:06:39 2024 2024-09-17T23:06:39.904 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:06:39.904 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:06:39.904 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: d6dfe272-2e04-4592-a96a-96648936ad18 2024-09-17T23:06:39.904 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: true 2024-09-17T23:06:39.907 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-17T23:06:39.907 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:06:39.907 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:06:39.908 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:06:39.908 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-17T23:06:39.908 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:06:39.908 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:06:39.908 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:06:39.908 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=started 2024-09-17T23:06:39.908 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:06:39.908 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:06:39.908 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:06:40.909 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-17T23:06:40.909 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:06:40.909 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:06:40.909 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:06:40.909 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=started 2024-09-17T23:06:40.909 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_result 2024-09-17T23:06:40.909 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=stopped 2024-09-17T23:06:40.910 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-17T23:06:40.910 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_instance=cluster1 2024-09-17T23:06:40.910 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:06:40.910 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ grep -i state 2024-09-17T23:06:40.910 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:06:40.911 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance=cluster1 2024-09-17T23:06:40.911 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local loop_instance 2024-09-17T23:06:40.911 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:06:40.911 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sleep 0 2024-09-17T23:06:40.911 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-17T23:06:40.911 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ seq 0 3 2024-09-17T23:06:40.914 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:06:40.914 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-17T23:06:40.914 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:0 2024-09-17T23:06:40.914 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:06:40.914 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:06:40.914 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-17T23:06:40.914 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:0 2024-09-17T23:06:40.914 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:06:40.915 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:06:40.915 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:06:40.915 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=0 2024-09-17T23:06:40.915 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-17T23:06:40.915 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:06:40.915 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:06:40.915 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=0 2024-09-17T23:06:40.915 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:06:40.915 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-17T23:06:40.915 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:0 2024-09-17T23:06:40.915 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:06:40.915 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:06:40.915 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-17T23:06:40.915 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:0 2024-09-17T23:06:40.915 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:06:40.916 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:06:40.916 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:06:40.916 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:06:40.916 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-17T23:06:40.916 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:06:40.916 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:06:40.916 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=0 2024-09-17T23:06:40.916 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=0 2024-09-17T23:06:40.916 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-17T23:06:40.936 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:06:40.936 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:06:40.936 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:06:40.936 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-17T23:06:41.016 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-17T23:06:41.016 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd mirror status : get status for rbd mirror 2024-09-17T23:06:41.016 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:admin_socket: invalid command 2024-09-17T23:06:41.019 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:06:41.019 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-17T23:06:41.019 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:1 2024-09-17T23:06:41.020 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:06:41.020 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:06:41.020 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-17T23:06:41.020 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:1 2024-09-17T23:06:41.020 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:06:41.020 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:06:41.020 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:06:41.020 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=1 2024-09-17T23:06:41.020 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-17T23:06:41.020 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:06:41.020 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:06:41.020 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=1 2024-09-17T23:06:41.020 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=1 2024-09-17T23:06:41.020 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-17T23:06:41.021 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:1 2024-09-17T23:06:41.021 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:06:41.021 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:06:41.021 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-17T23:06:41.021 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:1 2024-09-17T23:06:41.021 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:06:41.021 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:06:41.021 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:06:41.021 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=1 2024-09-17T23:06:41.021 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-17T23:06:41.021 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:06:41.021 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:06:41.021 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=1 2024-09-17T23:06:41.021 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=1 2024-09-17T23:06:41.022 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-17T23:06:41.040 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-17T23:06:41.040 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-17T23:06:41.040 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-17T23:06:41.040 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-17T23:06:41.113 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-17T23:06:41.114 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd mirror status : get status for rbd mirror 2024-09-17T23:06:41.114 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:admin_socket: invalid command 2024-09-17T23:06:41.117 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:06:41.117 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-17T23:06:41.117 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:2 2024-09-17T23:06:41.117 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:06:41.117 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:06:41.117 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-17T23:06:41.117 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:2 2024-09-17T23:06:41.117 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:06:41.118 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:06:41.118 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:06:41.118 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=2 2024-09-17T23:06:41.118 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-17T23:06:41.118 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:06:41.118 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:06:41.118 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=2 2024-09-17T23:06:41.118 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=2 2024-09-17T23:06:41.118 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-17T23:06:41.118 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:2 2024-09-17T23:06:41.118 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:06:41.118 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:06:41.118 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-17T23:06:41.118 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:2 2024-09-17T23:06:41.118 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:06:41.119 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:06:41.119 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:06:41.119 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=2 2024-09-17T23:06:41.119 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-17T23:06:41.119 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:06:41.119 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:06:41.119 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=2 2024-09-17T23:06:41.119 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=2 2024-09-17T23:06:41.119 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-17T23:06:41.137 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-17T23:06:41.137 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-17T23:06:41.137 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-17T23:06:41.137 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-17T23:06:41.210 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-17T23:06:41.210 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd mirror status : get status for rbd mirror 2024-09-17T23:06:41.210 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:admin_socket: invalid command 2024-09-17T23:06:41.213 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:06:41.213 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-09-17T23:06:41.213 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:3 2024-09-17T23:06:41.213 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:06:41.213 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:06:41.213 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-17T23:06:41.213 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:3 2024-09-17T23:06:41.214 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:06:41.214 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:06:41.214 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:06:41.214 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=3 2024-09-17T23:06:41.214 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-17T23:06:41.214 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:06:41.214 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:06:41.214 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=3 2024-09-17T23:06:41.214 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=3 2024-09-17T23:06:41.214 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-17T23:06:41.214 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:3 2024-09-17T23:06:41.214 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:06:41.214 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:06:41.214 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-17T23:06:41.214 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:3 2024-09-17T23:06:41.215 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:06:41.215 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:06:41.215 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:06:41.215 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=3 2024-09-17T23:06:41.215 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-17T23:06:41.215 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:06:41.215 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:06:41.215 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=3 2024-09-17T23:06:41.215 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=3 2024-09-17T23:06:41.215 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-17T23:06:41.232 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-17T23:06:41.232 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-17T23:06:41.232 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-17T23:06:41.232 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-09-17T23:06:41.306 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-17T23:06:41.307 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd mirror status : get status for rbd mirror 2024-09-17T23:06:41.307 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:admin_socket: invalid command 2024-09-17T23:06:41.307 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:06:41.308 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sleep 1 2024-09-17T23:06:42.309 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-17T23:06:42.309 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ seq 0 3 2024-09-17T23:06:42.312 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:06:42.312 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-17T23:06:42.312 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:0 2024-09-17T23:06:42.312 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:06:42.312 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:06:42.312 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-17T23:06:42.312 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:0 2024-09-17T23:06:42.312 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:06:42.312 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:06:42.312 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:06:42.312 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=0 2024-09-17T23:06:42.313 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-17T23:06:42.313 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:06:42.313 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:06:42.313 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=0 2024-09-17T23:06:42.313 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:06:42.313 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-17T23:06:42.313 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:0 2024-09-17T23:06:42.313 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:06:42.313 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:06:42.313 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-17T23:06:42.313 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:0 2024-09-17T23:06:42.313 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:06:42.313 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:06:42.313 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:06:42.313 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:06:42.314 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-17T23:06:42.314 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:06:42.314 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:06:42.314 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=0 2024-09-17T23:06:42.314 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=0 2024-09-17T23:06:42.314 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-17T23:06:42.334 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:06:42.334 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:06:42.334 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:06:42.335 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-17T23:06:42.415 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-17T23:06:42.415 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd mirror status : get status for rbd mirror 2024-09-17T23:06:42.415 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:admin_socket: invalid command 2024-09-17T23:06:42.417 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:06:42.417 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-17T23:06:42.417 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:1 2024-09-17T23:06:42.417 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:06:42.418 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:06:42.418 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-17T23:06:42.418 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:1 2024-09-17T23:06:42.418 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:06:42.418 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:06:42.418 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:06:42.418 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=1 2024-09-17T23:06:42.418 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-17T23:06:42.418 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:06:42.418 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:06:42.418 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=1 2024-09-17T23:06:42.418 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=1 2024-09-17T23:06:42.418 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-17T23:06:42.418 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:1 2024-09-17T23:06:42.419 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:06:42.419 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:06:42.419 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-17T23:06:42.419 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:1 2024-09-17T23:06:42.419 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:06:42.419 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:06:42.419 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:06:42.419 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=1 2024-09-17T23:06:42.419 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-17T23:06:42.419 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:06:42.419 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:06:42.419 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=1 2024-09-17T23:06:42.419 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=1 2024-09-17T23:06:42.419 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-17T23:06:42.437 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-17T23:06:42.437 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-17T23:06:42.437 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-17T23:06:42.437 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-17T23:06:42.510 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-17T23:06:42.511 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd mirror status : get status for rbd mirror 2024-09-17T23:06:42.511 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:admin_socket: invalid command 2024-09-17T23:06:42.513 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:06:42.513 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-17T23:06:42.513 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:2 2024-09-17T23:06:42.513 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:06:42.513 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:06:42.514 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-17T23:06:42.514 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:2 2024-09-17T23:06:42.514 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:06:42.514 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:06:42.514 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:06:42.514 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=2 2024-09-17T23:06:42.514 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-17T23:06:42.514 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:06:42.514 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:06:42.514 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=2 2024-09-17T23:06:42.514 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=2 2024-09-17T23:06:42.514 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-17T23:06:42.514 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:2 2024-09-17T23:06:42.514 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:06:42.515 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:06:42.515 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-17T23:06:42.515 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:2 2024-09-17T23:06:42.515 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:06:42.515 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:06:42.515 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:06:42.515 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=2 2024-09-17T23:06:42.515 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-17T23:06:42.515 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:06:42.515 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:06:42.515 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=2 2024-09-17T23:06:42.515 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=2 2024-09-17T23:06:42.515 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-17T23:06:42.532 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-17T23:06:42.532 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-17T23:06:42.533 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-17T23:06:42.533 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-17T23:06:42.605 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-17T23:06:42.605 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd mirror status : get status for rbd mirror 2024-09-17T23:06:42.606 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:admin_socket: invalid command 2024-09-17T23:06:42.608 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:06:42.608 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-09-17T23:06:42.609 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:3 2024-09-17T23:06:42.609 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:06:42.609 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:06:42.609 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-17T23:06:42.609 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:3 2024-09-17T23:06:42.609 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:06:42.609 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:06:42.609 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:06:42.609 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=3 2024-09-17T23:06:42.609 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-17T23:06:42.609 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:06:42.609 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:06:42.609 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=3 2024-09-17T23:06:42.609 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=3 2024-09-17T23:06:42.609 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-17T23:06:42.610 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:3 2024-09-17T23:06:42.610 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:06:42.610 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:06:42.610 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-17T23:06:42.610 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:3 2024-09-17T23:06:42.610 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:06:42.610 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:06:42.610 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:06:42.610 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=3 2024-09-17T23:06:42.610 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-17T23:06:42.610 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:06:42.610 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:06:42.610 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=3 2024-09-17T23:06:42.610 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=3 2024-09-17T23:06:42.610 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-17T23:06:42.627 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-17T23:06:42.627 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-17T23:06:42.627 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-17T23:06:42.628 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-09-17T23:06:42.703 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-17T23:06:42.703 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd mirror status : get status for rbd mirror 2024-09-17T23:06:42.703 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:admin_socket: invalid command 2024-09-17T23:06:42.704 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:06:42.704 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sleep 2 2024-09-17T23:06:44.705 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-17T23:06:44.706 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ seq 0 3 2024-09-17T23:06:44.708 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:06:44.709 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-17T23:06:44.709 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:0 2024-09-17T23:06:44.709 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:06:44.709 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:06:44.709 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-17T23:06:44.709 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:0 2024-09-17T23:06:44.709 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:06:44.709 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:06:44.709 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:06:44.709 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=0 2024-09-17T23:06:44.709 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-17T23:06:44.709 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:06:44.709 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:06:44.709 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=0 2024-09-17T23:06:44.710 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:06:44.710 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-17T23:06:44.710 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:0 2024-09-17T23:06:44.710 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:06:44.710 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:06:44.710 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-17T23:06:44.710 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:0 2024-09-17T23:06:44.710 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:06:44.710 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:06:44.710 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:06:44.710 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:06:44.710 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-17T23:06:44.710 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:06:44.711 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:06:44.711 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=0 2024-09-17T23:06:44.711 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=0 2024-09-17T23:06:44.711 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-17T23:06:44.731 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:06:44.731 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:06:44.731 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:06:44.731 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-17T23:06:44.811 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-17T23:06:44.811 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd mirror status : get status for rbd mirror 2024-09-17T23:06:44.812 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:admin_socket: invalid command 2024-09-17T23:06:44.815 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:06:44.815 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-17T23:06:44.815 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:1 2024-09-17T23:06:44.815 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:06:44.815 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:06:44.815 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-17T23:06:44.815 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:1 2024-09-17T23:06:44.815 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:06:44.815 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:06:44.815 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:06:44.815 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=1 2024-09-17T23:06:44.815 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-17T23:06:44.816 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:06:44.816 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:06:44.816 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=1 2024-09-17T23:06:44.816 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=1 2024-09-17T23:06:44.816 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-17T23:06:44.816 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:1 2024-09-17T23:06:44.816 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:06:44.816 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:06:44.816 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-17T23:06:44.816 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:1 2024-09-17T23:06:44.816 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:06:44.816 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:06:44.816 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:06:44.816 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=1 2024-09-17T23:06:44.817 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-17T23:06:44.817 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:06:44.817 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:06:44.817 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=1 2024-09-17T23:06:44.817 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=1 2024-09-17T23:06:44.817 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-17T23:06:44.834 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-17T23:06:44.834 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-17T23:06:44.834 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-17T23:06:44.834 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-17T23:06:44.908 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-17T23:06:44.908 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd mirror status : get status for rbd mirror 2024-09-17T23:06:44.908 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:admin_socket: invalid command 2024-09-17T23:06:44.911 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:06:44.911 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-17T23:06:44.911 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:2 2024-09-17T23:06:44.911 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:06:44.911 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:06:44.911 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-17T23:06:44.911 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:2 2024-09-17T23:06:44.911 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:06:44.911 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:06:44.911 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:06:44.911 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=2 2024-09-17T23:06:44.911 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-17T23:06:44.912 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:06:44.912 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:06:44.912 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=2 2024-09-17T23:06:44.912 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=2 2024-09-17T23:06:44.912 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-17T23:06:44.912 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:2 2024-09-17T23:06:44.912 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:06:44.912 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:06:44.912 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-17T23:06:44.912 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:2 2024-09-17T23:06:44.912 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:06:44.912 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:06:44.912 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:06:44.912 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=2 2024-09-17T23:06:44.913 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-17T23:06:44.913 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:06:44.913 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:06:44.913 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=2 2024-09-17T23:06:44.913 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=2 2024-09-17T23:06:44.913 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-17T23:06:44.931 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-17T23:06:44.931 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-17T23:06:44.931 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-17T23:06:44.931 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-17T23:06:45.004 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-17T23:06:45.004 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd mirror status : get status for rbd mirror 2024-09-17T23:06:45.004 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:admin_socket: invalid command 2024-09-17T23:06:45.007 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:06:45.007 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-09-17T23:06:45.007 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:3 2024-09-17T23:06:45.007 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:06:45.007 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:06:45.007 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-17T23:06:45.007 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:3 2024-09-17T23:06:45.007 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:06:45.007 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:06:45.007 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:06:45.007 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=3 2024-09-17T23:06:45.008 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-17T23:06:45.008 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:06:45.008 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:06:45.008 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=3 2024-09-17T23:06:45.008 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=3 2024-09-17T23:06:45.008 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-17T23:06:45.008 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:3 2024-09-17T23:06:45.008 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:06:45.008 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:06:45.008 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-17T23:06:45.008 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:3 2024-09-17T23:06:45.008 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:06:45.008 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:06:45.008 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:06:45.009 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=3 2024-09-17T23:06:45.009 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-17T23:06:45.009 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:06:45.009 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:06:45.009 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=3 2024-09-17T23:06:45.009 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=3 2024-09-17T23:06:45.009 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-17T23:06:45.027 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-17T23:06:45.027 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-17T23:06:45.027 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-17T23:06:45.027 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-09-17T23:06:45.099 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-17T23:06:45.099 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd mirror status : get status for rbd mirror 2024-09-17T23:06:45.099 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:admin_socket: invalid command 2024-09-17T23:06:45.100 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:06:45.101 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sleep 4 2024-09-17T23:06:49.102 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-17T23:06:49.102 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ seq 0 3 2024-09-17T23:06:49.105 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:06:49.105 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-17T23:06:49.105 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:0 2024-09-17T23:06:49.105 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:06:49.105 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:06:49.105 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-17T23:06:49.106 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:0 2024-09-17T23:06:49.106 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:06:49.106 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:06:49.106 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:06:49.106 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=0 2024-09-17T23:06:49.106 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-17T23:06:49.106 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:06:49.106 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:06:49.106 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=0 2024-09-17T23:06:49.106 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:06:49.106 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-17T23:06:49.106 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:0 2024-09-17T23:06:49.106 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:06:49.106 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:06:49.106 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-17T23:06:49.107 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:0 2024-09-17T23:06:49.107 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:06:49.107 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:06:49.107 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:06:49.107 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:06:49.107 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-17T23:06:49.107 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:06:49.107 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:06:49.107 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=0 2024-09-17T23:06:49.107 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=0 2024-09-17T23:06:49.107 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-17T23:06:49.127 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:06:49.128 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:06:49.128 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:06:49.128 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-17T23:06:49.208 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-17T23:06:49.208 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd mirror status : get status for rbd mirror 2024-09-17T23:06:49.208 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:admin_socket: invalid command 2024-09-17T23:06:49.211 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:06:49.211 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-17T23:06:49.211 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:1 2024-09-17T23:06:49.212 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:06:49.212 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:06:49.212 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-17T23:06:49.212 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:1 2024-09-17T23:06:49.212 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:06:49.212 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:06:49.212 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:06:49.212 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=1 2024-09-17T23:06:49.212 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-17T23:06:49.212 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:06:49.212 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:06:49.212 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=1 2024-09-17T23:06:49.213 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=1 2024-09-17T23:06:49.213 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-17T23:06:49.213 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:1 2024-09-17T23:06:49.213 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:06:49.213 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:06:49.213 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-17T23:06:49.213 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:1 2024-09-17T23:06:49.213 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:06:49.213 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:06:49.213 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:06:49.213 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=1 2024-09-17T23:06:49.213 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-17T23:06:49.213 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:06:49.214 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:06:49.214 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=1 2024-09-17T23:06:49.214 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=1 2024-09-17T23:06:49.214 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-17T23:06:49.231 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-17T23:06:49.231 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-17T23:06:49.231 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-17T23:06:49.231 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-17T23:06:49.310 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-17T23:06:49.310 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd mirror status : get status for rbd mirror 2024-09-17T23:06:49.310 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:admin_socket: invalid command 2024-09-17T23:06:49.313 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:06:49.313 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-17T23:06:49.313 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:2 2024-09-17T23:06:49.313 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:06:49.313 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:06:49.313 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-17T23:06:49.313 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:2 2024-09-17T23:06:49.313 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:06:49.314 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:06:49.314 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:06:49.314 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=2 2024-09-17T23:06:49.314 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-17T23:06:49.314 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:06:49.314 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:06:49.314 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=2 2024-09-17T23:06:49.314 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=2 2024-09-17T23:06:49.314 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-17T23:06:49.314 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:2 2024-09-17T23:06:49.314 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:06:49.314 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:06:49.314 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-17T23:06:49.315 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:2 2024-09-17T23:06:49.315 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:06:49.315 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:06:49.315 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:06:49.315 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=2 2024-09-17T23:06:49.315 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-17T23:06:49.315 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:06:49.315 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:06:49.315 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=2 2024-09-17T23:06:49.315 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=2 2024-09-17T23:06:49.315 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-17T23:06:49.332 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-17T23:06:49.332 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-17T23:06:49.332 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-17T23:06:49.332 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-17T23:06:49.405 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-17T23:06:49.405 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd mirror status : get status for rbd mirror 2024-09-17T23:06:49.406 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:admin_socket: invalid command 2024-09-17T23:06:49.409 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:06:49.409 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-09-17T23:06:49.409 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:3 2024-09-17T23:06:49.409 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:06:49.409 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:06:49.409 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-17T23:06:49.409 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:3 2024-09-17T23:06:49.409 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:06:49.409 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:06:49.409 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:06:49.409 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=3 2024-09-17T23:06:49.409 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-17T23:06:49.410 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:06:49.410 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:06:49.410 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=3 2024-09-17T23:06:49.410 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=3 2024-09-17T23:06:49.410 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-17T23:06:49.410 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:3 2024-09-17T23:06:49.410 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:06:49.410 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:06:49.410 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-17T23:06:49.410 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:3 2024-09-17T23:06:49.410 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:06:49.410 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:06:49.410 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:06:49.410 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=3 2024-09-17T23:06:49.410 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-17T23:06:49.411 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:06:49.411 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:06:49.411 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=3 2024-09-17T23:06:49.411 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=3 2024-09-17T23:06:49.411 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-17T23:06:49.427 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-17T23:06:49.427 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-17T23:06:49.427 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-17T23:06:49.427 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-09-17T23:06:49.500 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-17T23:06:49.501 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd mirror status : get status for rbd mirror 2024-09-17T23:06:49.501 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:admin_socket: invalid command 2024-09-17T23:06:49.501 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:06:49.502 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sleep 8 2024-09-17T23:06:57.503 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-17T23:06:57.503 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ seq 0 3 2024-09-17T23:06:57.506 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:06:57.506 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-17T23:06:57.506 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:0 2024-09-17T23:06:57.506 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:06:57.506 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:06:57.506 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-17T23:06:57.506 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:0 2024-09-17T23:06:57.507 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:06:57.507 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:06:57.507 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:06:57.507 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=0 2024-09-17T23:06:57.507 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-17T23:06:57.507 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:06:57.507 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:06:57.507 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=0 2024-09-17T23:06:57.507 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:06:57.507 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-17T23:06:57.507 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:0 2024-09-17T23:06:57.507 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:06:57.508 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:06:57.508 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-17T23:06:57.508 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:0 2024-09-17T23:06:57.508 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:06:57.508 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:06:57.508 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:06:57.508 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:06:57.508 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-17T23:06:57.508 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:06:57.508 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:06:57.508 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=0 2024-09-17T23:06:57.508 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=0 2024-09-17T23:06:57.508 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-17T23:06:57.530 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:06:57.530 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:06:57.530 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:06:57.530 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-17T23:06:57.621 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ return 0 2024-09-17T23:06:57.622 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ status_result=' "state": "Replaying"' 2024-09-17T23:06:57.622 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ echo ' "state": "Replaying"' 2024-09-17T23:06:57.622 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep -i Replaying 2024-09-17T23:06:57.623 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: "state": "Replaying" 2024-09-17T23:06:57.623 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_state=started 2024-09-17T23:06:57.624 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test started = started 2024-09-17T23:06:57.624 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:06:57.624 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ clean_snap_name= 2024-09-17T23:06:57.624 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ seq 1 10 2024-09-17T23:06:57.625 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 10` 2024-09-17T23:06:57.625 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ stress_write_image cluster2 mirror test 2024-09-17T23:06:57.625 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:06:57.625 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:06:57.625 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:06:57.625 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-17T23:06:57.626 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local duration=14 2024-09-17T23:06:57.626 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set +e 2024-09-17T23:06:57.627 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-17T23:06:57.761 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:100 IOs, 15537152 bytes 2024-09-17T23:06:57.855 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:200 IOs, 71611392 bytes 2024-09-17T23:06:57.953 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:300 IOs, 137648128 bytes 2024-09-17T23:06:58.005 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:400 IOs, 151428096 bytes 2024-09-17T23:06:58.117 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:500 IOs, 253280256 bytes 2024-09-17T23:06:58.160 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:600 IOs, 253899776 bytes 2024-09-17T23:06:58.242 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:700 IOs, 311470080 bytes 2024-09-17T23:06:58.294 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:800 IOs, 335028224 bytes 2024-09-17T23:06:58.370 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:900 IOs, 383028224 bytes 2024-09-17T23:06:58.427 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1000 IOs, 405956608 bytes 2024-09-17T23:06:58.493 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1100 IOs, 442798080 bytes 2024-09-17T23:06:58.582 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1200 IOs, 509158400 bytes 2024-09-17T23:06:58.663 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1300 IOs, 561719296 bytes 2024-09-17T23:06:58.717 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1400 IOs, 581002240 bytes 2024-09-17T23:06:58.773 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1500 IOs, 606220288 bytes 2024-09-17T23:06:58.966 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1600 IOs, 789046272 bytes 2024-09-17T23:06:59.052 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1700 IOs, 833842176 bytes 2024-09-17T23:06:59.109 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1800 IOs, 869600256 bytes 2024-09-17T23:06:59.177 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1900 IOs, 886505472 bytes 2024-09-17T23:06:59.254 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2000 IOs, 929256448 bytes 2024-09-17T23:06:59.331 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2100 IOs, 972827648 bytes 2024-09-17T23:06:59.439 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2200 IOs, 1051977728 bytes 2024-09-17T23:06:59.538 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2300 IOs, 1126431744 bytes 2024-09-17T23:06:59.605 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2400 IOs, 1147285504 bytes 2024-09-17T23:06:59.748 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2500 IOs, 1257995264 bytes 2024-09-17T23:06:59.817 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2600 IOs, 1291714560 bytes 2024-09-17T23:06:59.929 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2700 IOs, 1375314944 bytes 2024-09-17T23:07:00.028 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2800 IOs, 1441477632 bytes 2024-09-17T23:07:00.164 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2900 IOs, 1543804928 bytes 2024-09-17T23:07:00.251 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3000 IOs, 1594768384 bytes 2024-09-17T23:07:00.356 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3100 IOs, 1664558080 bytes 2024-09-17T23:07:00.427 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3200 IOs, 1692355584 bytes 2024-09-17T23:07:00.540 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3300 IOs, 1772076032 bytes 2024-09-17T23:07:00.614 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3400 IOs, 1831237632 bytes 2024-09-17T23:07:00.691 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3500 IOs, 1848851456 bytes 2024-09-17T23:07:00.778 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3600 IOs, 1904964608 bytes 2024-09-17T23:07:00.896 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3700 IOs, 1992132608 bytes 2024-09-17T23:07:00.976 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3800 IOs, 2029920256 bytes 2024-09-17T23:07:01.045 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3900 IOs, 2050232320 bytes 2024-09-17T23:07:01.145 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4000 IOs, 2113984512 bytes 2024-09-17T23:07:01.259 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4100 IOs, 2182824960 bytes 2024-09-17T23:07:01.369 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4200 IOs, 2226729984 bytes 2024-09-17T23:07:02.066 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4300 IOs, 2293334016 bytes 2024-09-17T23:07:02.768 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi038.stderr:2024-09-17T23:07:02.766+0000 7f5a4ac0c640 -1 rbd::mirror::Mirror: 0x55dc1f94fe60 update_pool_replayers: restarting failed pool replayer for uuid: 803aac29-dea2-475c-a073-828f4b7740a3 cluster: cluster2 client: client.mirror 2024-09-17T23:07:03.041 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi038.stderr:2024-09-17T23:07:03.038+0000 7f5a4ac0c640 -1 rbd::mirror::Mirror: 0x55dc1f94fe60 update_pool_replayers: restarting failed pool replayer for uuid: f5a8ac7c-03fd-4f9c-8233-cb887578c29b cluster: cluster2 client: client.mirror 2024-09-17T23:07:03.050 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4400 IOs, 2318349312 bytes 2024-09-17T23:07:03.215 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4500 IOs, 2457838592 bytes 2024-09-17T23:07:03.296 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4600 IOs, 2485578752 bytes 2024-09-17T23:07:03.360 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4700 IOs, 2507615232 bytes 2024-09-17T23:07:03.427 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4800 IOs, 2536096768 bytes 2024-09-17T23:07:03.547 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4900 IOs, 2638424064 bytes 2024-09-17T23:07:03.636 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5000 IOs, 2672744448 bytes 2024-09-17T23:07:03.718 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5100 IOs, 2713828352 bytes 2024-09-17T23:07:03.817 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5200 IOs, 2796307456 bytes 2024-09-17T23:07:03.913 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5300 IOs, 2833591296 bytes 2024-09-17T23:07:03.981 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5400 IOs, 2878060544 bytes 2024-09-17T23:07:04.068 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5500 IOs, 2897968128 bytes 2024-09-17T23:07:04.134 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5600 IOs, 2923633664 bytes 2024-09-17T23:07:04.281 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5700 IOs, 3021753344 bytes 2024-09-17T23:07:05.078 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5800 IOs, 3151520768 bytes 2024-09-17T23:07:05.177 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5900 IOs, 3212969984 bytes 2024-09-17T23:07:05.290 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6000 IOs, 3290432512 bytes 2024-09-17T23:07:05.395 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6100 IOs, 3356420096 bytes 2024-09-17T23:07:05.463 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6200 IOs, 3399093248 bytes 2024-09-17T23:07:05.551 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6300 IOs, 3434901504 bytes 2024-09-17T23:07:05.640 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6400 IOs, 3487870976 bytes 2024-09-17T23:07:05.709 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6500 IOs, 3513548800 bytes 2024-09-17T23:07:05.772 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6600 IOs, 3561676800 bytes 2024-09-17T23:07:05.890 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6700 IOs, 3620714496 bytes 2024-09-17T23:07:05.951 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6800 IOs, 3641084928 bytes 2024-09-17T23:07:06.003 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6900 IOs, 3663285248 bytes 2024-09-17T23:07:06.077 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7000 IOs, 3687454720 bytes 2024-09-17T23:07:06.170 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7100 IOs, 3744972800 bytes 2024-09-17T23:07:06.227 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7200 IOs, 3759760384 bytes 2024-09-17T23:07:06.318 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7300 IOs, 3813918720 bytes 2024-09-17T23:07:06.390 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7400 IOs, 3848580096 bytes 2024-09-17T23:07:06.445 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7500 IOs, 3874652160 bytes 2024-09-17T23:07:06.536 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7600 IOs, 3913651200 bytes 2024-09-17T23:07:06.581 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7700 IOs, 3914305536 bytes 2024-09-17T23:07:06.657 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7800 IOs, 3951484928 bytes 2024-09-17T23:07:06.702 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7900 IOs, 3952077824 bytes 2024-09-17T23:07:06.769 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8000 IOs, 3977795584 bytes 2024-09-17T23:07:06.884 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8100 IOs, 4082684928 bytes 2024-09-17T23:07:06.974 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8200 IOs, 4131607552 bytes 2024-09-17T23:07:07.096 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8300 IOs, 4203631616 bytes 2024-09-17T23:07:07.145 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8400 IOs, 4210806784 bytes 2024-09-17T23:07:07.206 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8500 IOs, 4254817280 bytes 2024-09-17T23:07:07.310 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8600 IOs, 4300903424 bytes 2024-09-17T23:07:07.380 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8700 IOs, 4332538880 bytes 2024-09-17T23:07:07.450 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8800 IOs, 4363782144 bytes 2024-09-17T23:07:07.511 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8900 IOs, 4385606656 bytes 2024-09-17T23:07:07.571 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9000 IOs, 4404209664 bytes 2024-09-17T23:07:07.657 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9100 IOs, 4465482752 bytes 2024-09-17T23:07:07.806 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9200 IOs, 4560425984 bytes 2024-09-17T23:07:07.937 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9300 IOs, 4681515008 bytes 2024-09-17T23:07:08.005 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9400 IOs, 4687611904 bytes 2024-09-17T23:07:08.055 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9500 IOs, 4695332864 bytes 2024-09-17T23:07:08.155 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9600 IOs, 4756232192 bytes 2024-09-17T23:07:08.235 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9700 IOs, 4796063744 bytes 2024-09-17T23:07:08.309 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9800 IOs, 4834955264 bytes 2024-09-17T23:07:08.386 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9900 IOs, 4874579968 bytes 2024-09-17T23:07:08.522 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10000 IOs, 4989950976 bytes 2024-09-17T23:07:08.576 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10100 IOs, 4993886208 bytes 2024-09-17T23:07:08.690 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10200 IOs, 5083133952 bytes 2024-09-17T23:07:08.756 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10300 IOs, 5113618432 bytes 2024-09-17T23:07:08.839 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10400 IOs, 5150020608 bytes 2024-09-17T23:07:08.914 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10500 IOs, 5188039680 bytes 2024-09-17T23:07:09.190 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10600 IOs, 5225349120 bytes 2024-09-17T23:07:09.265 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10700 IOs, 5267993600 bytes 2024-09-17T23:07:09.342 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10800 IOs, 5298867200 bytes 2024-09-17T23:07:09.392 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10900 IOs, 5312541696 bytes 2024-09-17T23:07:09.504 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:11000 IOs, 5386414080 bytes 2024-09-17T23:07:09.552 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:11100 IOs, 5390471168 bytes 2024-09-17T23:07:09.666 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:11200 IOs, 5494187008 bytes 2024-09-17T23:07:09.761 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:11300 IOs, 5529913344 bytes 2024-09-17T23:07:09.822 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:11400 IOs, 5550521344 bytes 2024-09-17T23:07:09.915 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:11500 IOs, 5608558592 bytes 2024-09-17T23:07:10.000 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:11600 IOs, 5685637120 bytes 2024-09-17T23:07:10.097 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:11700 IOs, 5705345024 bytes 2024-09-17T23:07:10.161 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:11800 IOs, 5731929088 bytes 2024-09-17T23:07:10.244 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:11900 IOs, 5775241216 bytes 2024-09-17T23:07:10.346 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:12000 IOs, 5844690944 bytes 2024-09-17T23:07:10.520 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:12100 IOs, 5997436928 bytes 2024-09-17T23:07:10.628 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:12200 IOs, 6062038016 bytes 2024-09-17T23:07:10.694 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:12300 IOs, 6089237504 bytes 2024-09-17T23:07:10.871 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:12400 IOs, 6160611328 bytes 2024-09-17T23:07:10.937 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:12500 IOs, 6187879424 bytes 2024-09-17T23:07:11.041 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:12600 IOs, 6256651264 bytes 2024-09-17T23:07:11.106 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:12700 IOs, 6280058880 bytes 2024-09-17T23:07:11.193 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:12800 IOs, 6340307968 bytes 2024-09-17T23:07:11.244 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:12900 IOs, 6340912128 bytes 2024-09-17T23:07:11.301 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:13000 IOs, 6358108160 bytes 2024-09-17T23:07:11.361 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:13100 IOs, 6378510336 bytes 2024-09-17T23:07:11.408 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:13200 IOs, 6380289024 bytes 2024-09-17T23:07:11.462 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:13300 IOs, 6407434240 bytes 2024-09-17T23:07:11.574 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:13400 IOs, 6474275840 bytes 2024-09-17T23:07:11.632 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ error_code=124 2024-09-17T23:07:11.632 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set -e 2024-09-17T23:07:11.632 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' 124 -eq 124 ']' 2024-09-17T23:07:11.632 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:07:11.632 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-17T23:07:11.632 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:07:11.632 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:07:11.632 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:07:11.633 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state_pattern=up+replaying 2024-09-17T23:07:11.633 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local description_pattern= 2024-09-17T23:07:11.633 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local service_pattern= 2024-09-17T23:07:11.633 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:07:11.633 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:07:12.633 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-17T23:07:12.634 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:07:12.634 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:07:12.634 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:07:12.634 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state_pattern=up+replaying 2024-09-17T23:07:12.634 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local description_pattern= 2024-09-17T23:07:12.634 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local service_pattern= 2024-09-17T23:07:12.634 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status 2024-09-17T23:07:12.634 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ CEPH_ARGS= 2024-09-17T23:07:12.634 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-17T23:07:12.689 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ status='test: 2024-09-17T23:07:12.689 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: global_id: d6dfe272-2e04-4592-a96a-96648936ad18 2024-09-17T23:07:12.690 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: state: down+unknown 2024-09-17T23:07:12.690 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: description: status not found 2024-09-17T23:07:12.690 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: last_update: 2024-09-17T23:07:12.690 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: peer_sites: 2024-09-17T23:07:12.690 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: name: cluster2 2024-09-17T23:07:12.690 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: state: down+unknown 2024-09-17T23:07:12.690 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: description: status not found 2024-09-17T23:07:12.690 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: last_update: ' 2024-09-17T23:07:12.690 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep '^ state: .*up+replaying' 2024-09-17T23:07:12.691 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 1 2024-09-17T23:07:12.691 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:07:12.691 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 2 2024-09-17T23:07:14.693 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-17T23:07:14.693 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:07:14.693 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:07:14.693 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:07:14.693 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state_pattern=up+replaying 2024-09-17T23:07:14.693 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local description_pattern= 2024-09-17T23:07:14.693 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local service_pattern= 2024-09-17T23:07:14.693 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status 2024-09-17T23:07:14.693 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ CEPH_ARGS= 2024-09-17T23:07:14.693 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-17T23:07:14.751 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ status='test: 2024-09-17T23:07:14.751 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: global_id: d6dfe272-2e04-4592-a96a-96648936ad18 2024-09-17T23:07:14.752 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: state: down+unknown 2024-09-17T23:07:14.752 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: description: status not found 2024-09-17T23:07:14.752 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: last_update: 2024-09-17T23:07:14.752 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: peer_sites: 2024-09-17T23:07:14.752 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: name: cluster2 2024-09-17T23:07:14.752 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: state: down+unknown 2024-09-17T23:07:14.752 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: description: status not found 2024-09-17T23:07:14.752 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: last_update: ' 2024-09-17T23:07:14.752 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep '^ state: .*up+replaying' 2024-09-17T23:07:14.753 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 1 2024-09-17T23:07:14.753 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:07:14.753 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 4 2024-09-17T23:07:17.776 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.7581303583396939) > weight (0.25) 2024-09-17T23:07:17.776 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.7801789787177044) > weight (0.25) 2024-09-17T23:07:17.777 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.7596427614595802) > weight (0.25) 2024-09-17T23:07:17.777 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.3 2024-09-17T23:07:17.777 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Sent signal 15 2024-09-17T23:07:17.777 INFO:tasks.rbd_mirror_thrash:waiting for 9 secs before reviving daemons 2024-09-17T23:07:17.778 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:07:17.776+0000 7fe7d185d640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 36737) UID: 1000 2024-09-17T23:07:18.754 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-17T23:07:18.755 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:07:18.755 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:07:18.755 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:07:18.755 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state_pattern=up+replaying 2024-09-17T23:07:18.755 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local description_pattern= 2024-09-17T23:07:18.755 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local service_pattern= 2024-09-17T23:07:18.755 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status 2024-09-17T23:07:18.755 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ CEPH_ARGS= 2024-09-17T23:07:18.755 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-17T23:07:18.810 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ status='test: 2024-09-17T23:07:18.810 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: global_id: d6dfe272-2e04-4592-a96a-96648936ad18 2024-09-17T23:07:18.810 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: state: down+unknown 2024-09-17T23:07:18.810 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: description: status not found 2024-09-17T23:07:18.810 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: last_update: 2024-09-17T23:07:18.810 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: peer_sites: 2024-09-17T23:07:18.810 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: name: cluster2 2024-09-17T23:07:18.810 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: state: down+unknown 2024-09-17T23:07:18.810 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: description: status not found 2024-09-17T23:07:18.810 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: last_update: ' 2024-09-17T23:07:18.811 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep '^ state: .*up+replaying' 2024-09-17T23:07:18.811 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 1 2024-09-17T23:07:18.811 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:07:18.811 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 8 2024-09-17T23:07:26.778 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.3 2024-09-17T23:07:26.778 INFO:teuthology.orchestra.run:waiting for 600 2024-09-17T23:07:26.779 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.3 2024-09-17T23:07:26.779 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2024-09-17T23:07:26.779 DEBUG:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2024-09-17T23:07:26.782 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2024-09-17T23:07:26.783 INFO:tasks.rbd_mirror_thrash:waiting for 101 secs before thrashing 2024-09-17T23:07:26.813 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-17T23:07:26.813 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:07:26.813 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:07:26.813 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:07:26.813 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state_pattern=up+replaying 2024-09-17T23:07:26.813 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local description_pattern= 2024-09-17T23:07:26.814 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local service_pattern= 2024-09-17T23:07:26.814 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status 2024-09-17T23:07:26.814 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ CEPH_ARGS= 2024-09-17T23:07:26.814 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-17T23:07:26.867 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ status='test: 2024-09-17T23:07:26.867 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: global_id: d6dfe272-2e04-4592-a96a-96648936ad18 2024-09-17T23:07:26.867 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: state: up+replaying 2024-09-17T23:07:26.867 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726614399,"remote_snapshot_timestamp":1726614399,"replay_state":"idle"} 2024-09-17T23:07:26.868 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: service: mirror.0 on smithi038 2024-09-17T23:07:26.868 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: last_update: 2024-09-17 23:07:25 2024-09-17T23:07:26.868 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: peer_sites: 2024-09-17T23:07:26.868 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: name: cluster2 2024-09-17T23:07:26.868 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: state: up+stopped 2024-09-17T23:07:26.868 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: description: local image is primary 2024-09-17T23:07:26.868 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: last_update: 2024-09-17 23:07:24' 2024-09-17T23:07:26.868 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep '^ state: .*up+replaying' 2024-09-17T23:07:26.868 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: state: up+replaying 2024-09-17T23:07:26.869 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep '^ description: .*' 2024-09-17T23:07:26.869 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726614399,"remote_snapshot_timestamp":1726614399,"replay_state":"idle"} 2024-09-17T23:07:26.870 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n '' ']' 2024-09-17T23:07:26.870 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ echo up+replaying 2024-09-17T23:07:26.870 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep '^up+' 2024-09-17T23:07:26.871 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:up+replaying 2024-09-17T23:07:26.871 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:07:26.872 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: service: mirror.0 on smithi038 2024-09-17T23:07:26.873 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local last_update 2024-09-17T23:07:26.873 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:07:26.874 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ last_update='2024-09-17 23:07:25' 2024-09-17T23:07:26.875 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-17 23:07:25' 2024-09-17T23:07:26.875 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:07:26.875 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:07:26.875 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:07:26.875 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state_pattern=up+replaying 2024-09-17T23:07:26.875 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local 'prev_last_update=2024-09-17 23:07:25' 2024-09-17T23:07:26.875 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status 2024-09-17T23:07:26.875 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ CEPH_ARGS= 2024-09-17T23:07:26.875 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-17T23:07:26.939 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ status='OKOKOK142134295mirror.0smithi03819.1.1-330-gc2ddf972trueOK42214298mirror.2smithi03819.1.1-330-gc2ddf972falseOK42234294mirror.3smithi03819.1.1-330-gc2ddf972falseOK42624310mirror.1smithi03819.1.1-330-gc2ddf972falseOKtestd6dfe272-2e04-4592-a96a-96648936ad18up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726614399,"remote_snapshot_timestamp":1726614399,"replay_state":"idle"}42134295mirror.0smithi0382024-09-17 23:07:25cluster20ba5654d-47fb-4672-b20c-7359a1340314up+stoppedlocal image is primary2024-09-17 23:07:24' 2024-09-17T23:07:26.939 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local last_update state 2024-09-17T23:07:26.939 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-17T23:07:26.979 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ last_update='2024-09-17 23:07:25' 2024-09-17T23:07:26.979 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-17T23:07:26.981 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ state=up+replaying 2024-09-17T23:07:26.981 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ echo up+replaying 2024-09-17T23:07:26.981 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep up+replaying 2024-09-17T23:07:26.982 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:up+replaying 2024-09-17T23:07:26.983 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:07:26.983 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:07:26.983 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ snap_name=snap1 2024-09-17T23:07:26.983 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_snap cluster2 mirror test snap1 2024-09-17T23:07:26.983 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:07:26.983 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:07:26.983 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:07:26.983 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap1 2024-09-17T23:07:26.983 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap1 --debug-rbd=20 --debug-journaler=20 2024-09-17T23:07:32.781 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-09-17T23:07:32.782 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-09-17T23:07:32.782 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2024-09-17T23:07:32.783 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi038.stderr:2024-09-17T23:07:32.780+0000 7ffaaed6f640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 36625) UID: 1000 2024-09-17T23:07:35.783 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-09-17T23:07:35.783 INFO:teuthology.orchestra.run:waiting for 600 2024-09-17T23:07:35.783 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-09-17T23:07:35.783 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-09-17T23:07:35.784 DEBUG:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-09-17T23:07:35.786 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-09-17T23:07:35.787 INFO:tasks.rbd_mirror_thrash:waiting for 109 secs before thrashing 2024-09-17T23:07:42.674 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-17T23:07:42.674 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:07:42.674 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:07:42.674 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:07:42.674 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-17T23:07:42.674 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:07:42.674 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:07:42.674 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:07:42.674 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=started 2024-09-17T23:07:42.675 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:07:42.675 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:07:42.675 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:07:43.675 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-17T23:07:43.675 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:07:43.675 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:07:43.675 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:07:43.675 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=started 2024-09-17T23:07:43.676 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_result 2024-09-17T23:07:43.676 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=stopped 2024-09-17T23:07:43.677 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-17T23:07:43.677 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_instance=cluster1 2024-09-17T23:07:43.677 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:07:43.677 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ grep -i state 2024-09-17T23:07:43.677 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:07:43.677 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance=cluster1 2024-09-17T23:07:43.677 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local loop_instance 2024-09-17T23:07:43.677 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:07:43.678 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sleep 0 2024-09-17T23:07:43.678 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-17T23:07:43.678 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ seq 0 3 2024-09-17T23:07:43.680 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:07:43.680 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-17T23:07:43.680 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:0 2024-09-17T23:07:43.680 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:07:43.680 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:07:43.680 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-17T23:07:43.680 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:0 2024-09-17T23:07:43.680 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:07:43.680 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:07:43.680 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:07:43.680 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=0 2024-09-17T23:07:43.681 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-17T23:07:43.681 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:07:43.681 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:07:43.681 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=0 2024-09-17T23:07:43.681 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:07:43.681 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-17T23:07:43.681 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:0 2024-09-17T23:07:43.681 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:07:43.681 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:07:43.681 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-17T23:07:43.681 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:0 2024-09-17T23:07:43.681 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:07:43.681 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:07:43.681 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:07:43.682 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:07:43.682 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-17T23:07:43.682 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:07:43.683 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:07:43.683 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=0 2024-09-17T23:07:43.683 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=0 2024-09-17T23:07:43.683 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-17T23:07:43.702 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:07:43.702 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:07:43.702 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:07:43.702 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-17T23:07:43.785 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-17T23:07:43.785 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd mirror status : get status for rbd mirror 2024-09-17T23:07:43.785 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:admin_socket: invalid command 2024-09-17T23:07:43.788 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:07:43.788 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-17T23:07:43.788 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:1 2024-09-17T23:07:43.788 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:07:43.788 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:07:43.788 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-17T23:07:43.789 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:1 2024-09-17T23:07:43.789 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:07:43.789 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:07:43.789 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:07:43.789 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=1 2024-09-17T23:07:43.789 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-17T23:07:43.789 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:07:43.789 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:07:43.789 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=1 2024-09-17T23:07:43.789 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=1 2024-09-17T23:07:43.789 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-17T23:07:43.789 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:1 2024-09-17T23:07:43.789 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:07:43.789 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:07:43.790 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-17T23:07:43.790 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:1 2024-09-17T23:07:43.790 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:07:43.790 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:07:43.790 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:07:43.790 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=1 2024-09-17T23:07:43.790 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-17T23:07:43.790 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:07:43.790 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:07:43.790 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=1 2024-09-17T23:07:43.790 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=1 2024-09-17T23:07:43.790 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-17T23:07:43.808 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-17T23:07:43.808 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-17T23:07:43.808 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-17T23:07:43.808 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-17T23:07:43.885 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-17T23:07:43.885 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd mirror status : get status for rbd mirror 2024-09-17T23:07:43.885 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:admin_socket: invalid command 2024-09-17T23:07:43.887 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:07:43.887 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-17T23:07:43.888 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:2 2024-09-17T23:07:43.888 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:07:43.888 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:07:43.888 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-17T23:07:43.888 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:2 2024-09-17T23:07:43.888 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:07:43.888 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:07:43.888 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:07:43.888 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=2 2024-09-17T23:07:43.888 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-17T23:07:43.888 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:07:43.888 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:07:43.888 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=2 2024-09-17T23:07:43.888 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=2 2024-09-17T23:07:43.889 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-17T23:07:43.889 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:2 2024-09-17T23:07:43.889 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:07:43.889 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:07:43.889 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-17T23:07:43.889 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:2 2024-09-17T23:07:43.889 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:07:43.889 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:07:43.889 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:07:43.889 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=2 2024-09-17T23:07:43.889 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-17T23:07:43.889 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:07:43.889 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:07:43.889 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=2 2024-09-17T23:07:43.889 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=2 2024-09-17T23:07:43.890 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-17T23:07:43.906 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-17T23:07:43.906 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-17T23:07:43.906 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-17T23:07:43.907 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-17T23:07:43.982 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-17T23:07:43.982 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd mirror status : get status for rbd mirror 2024-09-17T23:07:43.982 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:admin_socket: invalid command 2024-09-17T23:07:43.985 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:07:43.985 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-09-17T23:07:43.985 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:3 2024-09-17T23:07:43.985 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:07:43.985 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:07:43.985 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-17T23:07:43.986 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:3 2024-09-17T23:07:43.986 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:07:43.986 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:07:43.986 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:07:43.986 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=3 2024-09-17T23:07:43.986 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-17T23:07:43.986 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:07:43.986 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:07:43.986 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=3 2024-09-17T23:07:43.986 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=3 2024-09-17T23:07:43.986 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-17T23:07:43.986 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:3 2024-09-17T23:07:43.986 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:07:43.986 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:07:43.986 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-17T23:07:43.987 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:3 2024-09-17T23:07:43.987 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:07:43.987 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:07:43.987 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:07:43.987 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=3 2024-09-17T23:07:43.987 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-17T23:07:43.987 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:07:43.987 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:07:43.987 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=3 2024-09-17T23:07:43.987 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=3 2024-09-17T23:07:43.987 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-17T23:07:44.004 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-17T23:07:44.004 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-17T23:07:44.004 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-17T23:07:44.004 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-09-17T23:07:44.088 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ return 0 2024-09-17T23:07:44.088 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ status_result=' "state": "Replaying"' 2024-09-17T23:07:44.089 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ echo ' "state": "Replaying"' 2024-09-17T23:07:44.089 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep -i Replaying 2024-09-17T23:07:44.089 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: "state": "Replaying" 2024-09-17T23:07:44.090 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_state=started 2024-09-17T23:07:44.091 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test started = started 2024-09-17T23:07:44.091 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:07:44.091 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-17T23:07:44.091 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:07:44.091 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:07:44.091 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:07:44.091 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:07:44.091 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = journal ']' 2024-09-17T23:07:44.091 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = snapshot ']' 2024-09-17T23:07:44.091 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-09-17T23:07:44.091 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:07:44.091 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:07:44.091 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:07:44.092 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:07:44.092 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster2-mirror-test.status 2024-09-17T23:07:44.092 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster2-mirror-test.status 2024-09-17T23:07:44.092 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:07:44.092 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-test.status 2024-09-17T23:07:44.093 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster1-mirror-test.status 2024-09-17T23:07:44.093 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster1-mirror-test.status 2024-09-17T23:07:44.093 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:07:44.093 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_status_log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-test.status 2024-09-17T23:07:44.094 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-17T23:07:44.094 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:07:44.094 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:07:44.094 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:07:44.094 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-17T23:07:44.672 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Snapshot ID: 5 2024-09-17T23:07:44.681 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.status 2024-09-17T23:07:44.681 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:07:44.681 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:07:44.681 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:07:44.681 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-test.status 2024-09-17T23:07:44.681 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-17T23:07:44.681 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:07:44.729 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-17T23:07:44.731 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snapshot_id=5 2024-09-17T23:07:44.731 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:07:44.731 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:07:44.731 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.2 2024-09-17T23:07:44.933 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.status 2024-09-17T23:07:44.933 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:07:44.933 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:07:44.933 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:07:44.933 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-test.status 2024-09-17T23:07:44.933 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-17T23:07:44.933 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:07:45.185 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:07:45.185 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:07:45.186 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:07:45.186 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:07:45.186 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:07:45.187 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:07:45.187 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 5 = '' 2024-09-17T23:07:45.187 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:07:45.187 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.4 2024-09-17T23:07:45.588 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.status 2024-09-17T23:07:45.588 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:07:45.589 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:07:45.589 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:07:45.589 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-test.status 2024-09-17T23:07:45.589 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-17T23:07:45.589 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:07:45.641 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:07:45.641 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:07:45.643 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:07:45.643 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:07:45.643 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:07:45.644 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:07:45.644 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 5 = '' 2024-09-17T23:07:45.644 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:07:45.644 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.8 2024-09-17T23:07:46.445 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.status 2024-09-17T23:07:46.445 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:07:46.446 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:07:46.446 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:07:46.446 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-test.status 2024-09-17T23:07:46.446 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-17T23:07:46.446 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:07:46.527 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:07:46.527 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:07:46.529 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:07:46.529 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:07:46.529 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:07:46.530 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:07:46.530 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 5 = '' 2024-09-17T23:07:46.530 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:07:46.530 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1.6 2024-09-17T23:07:48.131 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.status 2024-09-17T23:07:48.131 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:07:48.131 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:07:48.131 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:07:48.131 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-test.status 2024-09-17T23:07:48.131 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-17T23:07:48.132 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:07:48.184 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:07:48.187 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id=5 2024-09-17T23:07:48.187 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 5 = 5 2024-09-17T23:07:48.187 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:07:48.187 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snap_present cluster1 mirror test snap1 2024-09-17T23:07:48.187 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:07:48.187 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:07:48.187 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:07:48.188 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap1 2024-09-17T23:07:48.188 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:07:48.188 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-17T23:07:48.188 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:07:49.188 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 info mirror/test@snap1 2024-09-17T23:07:49.237 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'test': 2024-09-17T23:07:49.237 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 512 MiB in 128 objects 2024-09-17T23:07:49.237 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:07:49.237 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 2 2024-09-17T23:07:49.237 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 10c7e03f2edd 2024-09-17T23:07:49.238 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.10c7e03f2edd 2024-09-17T23:07:49.238 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:07:49.238 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-17T23:07:49.238 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:07:49.238 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:07:49.238 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:06:57 2024 2024-09-17T23:07:49.238 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:06:57 2024 2024-09-17T23:07:49.238 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:06:57 2024 2024-09-17T23:07:49.238 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: protected: False 2024-09-17T23:07:49.238 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:07:49.238 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:07:49.238 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: d6dfe272-2e04-4592-a96a-96648936ad18 2024-09-17T23:07:49.238 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: false 2024-09-17T23:07:49.241 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:07:49.242 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n '' ']' 2024-09-17T23:07:49.242 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ compare_image_snaps mirror test snap1 2024-09-17T23:07:49.242 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:07:49.242 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:07:49.242 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap1 2024-09-17T23:07:49.242 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local ret=0 2024-09-17T23:07:49.242 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local rmt_export=/tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export 2024-09-17T23:07:49.242 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local loc_export=/tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:07:49.242 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:07:49.242 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1 /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export 2024-09-17T23:07:49.763 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-17T23:07:49.774 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1 /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:07:50.488 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-17T23:07:50.498 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ cmp /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:07:50.743 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:07:50.861 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:07:50.862 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ clean_snap_name=snap1-clean 2024-09-17T23:07:50.862 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_snap cluster2 mirror test snap1-clean 2024-09-17T23:07:50.862 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:07:50.862 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:07:50.862 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:07:50.862 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap1-clean 2024-09-17T23:07:50.862 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap1-clean --debug-rbd=20 --debug-journaler=20 2024-09-17T23:07:51.694 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 10` 2024-09-17T23:07:51.694 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ stress_write_image cluster2 mirror test 2024-09-17T23:07:51.694 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:07:51.694 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:07:51.694 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:07:51.695 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-17T23:07:51.696 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local duration=13 2024-09-17T23:07:51.696 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set +e 2024-09-17T23:07:51.696 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-17T23:07:52.131 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:100 IOs, 25033728 bytes 2024-09-17T23:07:52.247 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:200 IOs, 77110272 bytes 2024-09-17T23:07:52.381 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:300 IOs, 127898624 bytes 2024-09-17T23:07:52.493 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:400 IOs, 203433984 bytes 2024-09-17T23:07:52.610 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:500 IOs, 283612160 bytes 2024-09-17T23:07:52.693 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:600 IOs, 324995072 bytes 2024-09-17T23:07:52.790 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:700 IOs, 379657216 bytes 2024-09-17T23:07:52.889 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:800 IOs, 433571840 bytes 2024-09-17T23:07:52.954 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:900 IOs, 454243328 bytes 2024-09-17T23:07:53.011 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1000 IOs, 461449216 bytes 2024-09-17T23:07:53.067 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1100 IOs, 474814464 bytes 2024-09-17T23:07:53.163 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1200 IOs, 533145600 bytes 2024-09-17T23:07:53.288 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1300 IOs, 626688000 bytes 2024-09-17T23:07:53.361 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1400 IOs, 671315968 bytes 2024-09-17T23:07:53.453 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1500 IOs, 713232384 bytes 2024-09-17T23:07:53.498 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1600 IOs, 713851904 bytes 2024-09-17T23:07:53.572 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1700 IOs, 746976256 bytes 2024-09-17T23:07:53.660 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1800 IOs, 807827456 bytes 2024-09-17T23:07:53.764 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1900 IOs, 860723200 bytes 2024-09-17T23:07:53.844 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2000 IOs, 898648064 bytes 2024-09-17T23:07:53.915 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2100 IOs, 945330176 bytes 2024-09-17T23:07:54.032 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2200 IOs, 1012974592 bytes 2024-09-17T23:07:54.117 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2300 IOs, 1058288640 bytes 2024-09-17T23:07:54.203 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2400 IOs, 1104077824 bytes 2024-09-17T23:07:54.294 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2500 IOs, 1153640448 bytes 2024-09-17T23:07:54.371 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2600 IOs, 1194469376 bytes 2024-09-17T23:07:54.461 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2700 IOs, 1251553280 bytes 2024-09-17T23:07:54.515 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2800 IOs, 1253388288 bytes 2024-09-17T23:07:54.621 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2900 IOs, 1324138496 bytes 2024-09-17T23:07:54.687 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3000 IOs, 1347678208 bytes 2024-09-17T23:07:54.762 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3100 IOs, 1381228544 bytes 2024-09-17T23:07:54.823 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3200 IOs, 1400295424 bytes 2024-09-17T23:07:54.935 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3300 IOs, 1473611776 bytes 2024-09-17T23:07:55.070 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3400 IOs, 1572540416 bytes 2024-09-17T23:07:55.187 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3500 IOs, 1655563264 bytes 2024-09-17T23:07:55.287 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3600 IOs, 1717652480 bytes 2024-09-17T23:07:55.384 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3700 IOs, 1769750528 bytes 2024-09-17T23:07:55.454 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3800 IOs, 1796680704 bytes 2024-09-17T23:07:55.689 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3900 IOs, 1797296128 bytes 2024-09-17T23:07:55.737 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4000 IOs, 1803047936 bytes 2024-09-17T23:07:55.783 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4100 IOs, 1803799552 bytes 2024-09-17T23:07:55.849 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4200 IOs, 1829125120 bytes 2024-09-17T23:07:55.922 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4300 IOs, 1861351424 bytes 2024-09-17T23:07:55.994 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4400 IOs, 1887937536 bytes 2024-09-17T23:07:56.144 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4500 IOs, 2007012352 bytes 2024-09-17T23:07:56.226 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4600 IOs, 2048796672 bytes 2024-09-17T23:07:56.286 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4700 IOs, 2063105024 bytes 2024-09-17T23:07:56.391 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4800 IOs, 2124934144 bytes 2024-09-17T23:07:56.459 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4900 IOs, 2149339136 bytes 2024-09-17T23:07:56.538 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5000 IOs, 2191985664 bytes 2024-09-17T23:07:56.608 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5100 IOs, 2218547200 bytes 2024-09-17T23:07:56.678 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5200 IOs, 2245256192 bytes 2024-09-17T23:07:56.747 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5300 IOs, 2296186880 bytes 2024-09-17T23:07:56.860 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5400 IOs, 2358396928 bytes 2024-09-17T23:07:56.933 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5500 IOs, 2384261120 bytes 2024-09-17T23:07:57.003 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5600 IOs, 2408939520 bytes 2024-09-17T23:07:57.110 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5700 IOs, 2497658880 bytes 2024-09-17T23:07:57.250 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5800 IOs, 2580997120 bytes 2024-09-17T23:07:57.321 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5900 IOs, 2621936640 bytes 2024-09-17T23:07:57.424 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6000 IOs, 2666609664 bytes 2024-09-17T23:07:57.488 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6100 IOs, 2682560512 bytes 2024-09-17T23:07:57.537 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6200 IOs, 2686819328 bytes 2024-09-17T23:07:57.621 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6300 IOs, 2730373120 bytes 2024-09-17T23:07:57.694 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6400 IOs, 2757772288 bytes 2024-09-17T23:07:57.751 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6500 IOs, 2772093952 bytes 2024-09-17T23:07:57.812 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6600 IOs, 2789757952 bytes 2024-09-17T23:07:57.859 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6700 IOs, 2802421760 bytes 2024-09-17T23:07:57.937 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6800 IOs, 2822202368 bytes 2024-09-17T23:07:58.177 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6900 IOs, 2855897088 bytes 2024-09-17T23:07:58.237 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7000 IOs, 2857762816 bytes 2024-09-17T23:07:58.341 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7100 IOs, 2923558912 bytes 2024-09-17T23:07:58.404 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7200 IOs, 2944778240 bytes 2024-09-17T23:07:58.483 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7300 IOs, 2975679488 bytes 2024-09-17T23:07:58.545 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7400 IOs, 2998356992 bytes 2024-09-17T23:07:58.616 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7500 IOs, 3032046592 bytes 2024-09-17T23:07:58.703 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7600 IOs, 3060800512 bytes 2024-09-17T23:07:58.786 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7700 IOs, 3110431744 bytes 2024-09-17T23:07:58.867 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7800 IOs, 3139580928 bytes 2024-09-17T23:07:58.929 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7900 IOs, 3160654848 bytes 2024-09-17T23:07:59.056 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8000 IOs, 3248942080 bytes 2024-09-17T23:07:59.138 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8100 IOs, 3285476352 bytes 2024-09-17T23:07:59.259 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8200 IOs, 3372362752 bytes 2024-09-17T23:07:59.330 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8300 IOs, 3393211392 bytes 2024-09-17T23:07:59.395 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8400 IOs, 3413894144 bytes 2024-09-17T23:07:59.504 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8500 IOs, 3481710592 bytes 2024-09-17T23:07:59.602 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8600 IOs, 3539019776 bytes 2024-09-17T23:07:59.670 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8700 IOs, 3564474368 bytes 2024-09-17T23:07:59.745 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8800 IOs, 3604537344 bytes 2024-09-17T23:07:59.799 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8900 IOs, 3605168128 bytes 2024-09-17T23:07:59.875 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9000 IOs, 3635097600 bytes 2024-09-17T23:08:00.008 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9100 IOs, 3734236160 bytes 2024-09-17T23:08:00.089 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9200 IOs, 3787256832 bytes 2024-09-17T23:08:00.227 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9300 IOs, 3872689152 bytes 2024-09-17T23:08:00.276 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9400 IOs, 3873324032 bytes 2024-09-17T23:08:00.324 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9500 IOs, 3874145280 bytes 2024-09-17T23:08:00.422 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9600 IOs, 3951669248 bytes 2024-09-17T23:08:00.512 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9700 IOs, 3998191616 bytes 2024-09-17T23:08:00.591 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9800 IOs, 4013628416 bytes 2024-09-17T23:08:00.662 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9900 IOs, 4039455744 bytes 2024-09-17T23:08:00.750 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10000 IOs, 4087214080 bytes 2024-09-17T23:08:00.846 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10100 IOs, 4135223296 bytes 2024-09-17T23:08:00.927 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10200 IOs, 4179688448 bytes 2024-09-17T23:08:01.003 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10300 IOs, 4201042944 bytes 2024-09-17T23:08:01.061 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10400 IOs, 4212454400 bytes 2024-09-17T23:08:01.108 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10500 IOs, 4213071872 bytes 2024-09-17T23:08:01.198 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10600 IOs, 4261464064 bytes 2024-09-17T23:08:01.274 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10700 IOs, 4293690368 bytes 2024-09-17T23:08:01.359 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10800 IOs, 4331181056 bytes 2024-09-17T23:08:01.443 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10900 IOs, 4373611520 bytes 2024-09-17T23:08:01.542 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:11000 IOs, 4429360128 bytes 2024-09-17T23:08:01.650 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:11100 IOs, 4497037312 bytes 2024-09-17T23:08:01.712 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:11200 IOs, 4511885312 bytes 2024-09-17T23:08:01.794 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:11300 IOs, 4549390336 bytes 2024-09-17T23:08:01.851 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:11400 IOs, 4567171072 bytes 2024-09-17T23:08:01.930 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:11500 IOs, 4597123072 bytes 2024-09-17T23:08:02.031 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:11600 IOs, 4661347328 bytes 2024-09-17T23:08:02.081 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:11700 IOs, 4663428096 bytes 2024-09-17T23:08:02.162 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:11800 IOs, 4706742272 bytes 2024-09-17T23:08:02.246 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:11900 IOs, 4739848192 bytes 2024-09-17T23:08:02.371 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:12000 IOs, 4831809536 bytes 2024-09-17T23:08:02.450 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:12100 IOs, 4858586112 bytes 2024-09-17T23:08:02.517 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:12200 IOs, 4879990784 bytes 2024-09-17T23:08:02.612 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:12300 IOs, 4934716416 bytes 2024-09-17T23:08:02.724 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:12400 IOs, 4998638592 bytes 2024-09-17T23:08:02.790 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:12500 IOs, 5019116544 bytes 2024-09-17T23:08:02.871 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:12600 IOs, 5062403072 bytes 2024-09-17T23:08:02.948 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:12700 IOs, 5089373184 bytes 2024-09-17T23:08:03.011 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:12800 IOs, 5109522432 bytes 2024-09-17T23:08:03.068 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:12900 IOs, 5121378304 bytes 2024-09-17T23:08:03.162 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:13000 IOs, 5168968704 bytes 2024-09-17T23:08:03.251 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:13100 IOs, 5214750720 bytes 2024-09-17T23:08:03.374 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:13200 IOs, 5294577664 bytes 2024-09-17T23:08:03.517 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:13300 IOs, 5407193088 bytes 2024-09-17T23:08:03.587 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:13400 IOs, 5428377600 bytes 2024-09-17T23:08:03.655 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:13500 IOs, 5455777792 bytes 2024-09-17T23:08:03.723 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:13600 IOs, 5471930368 bytes 2024-09-17T23:08:03.794 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:13700 IOs, 5501167616 bytes 2024-09-17T23:08:03.924 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:13800 IOs, 5593355264 bytes 2024-09-17T23:08:04.031 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:13900 IOs, 5662457856 bytes 2024-09-17T23:08:04.142 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:14000 IOs, 5732807680 bytes 2024-09-17T23:08:04.248 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:14100 IOs, 5802185728 bytes 2024-09-17T23:08:04.368 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:14200 IOs, 5879897088 bytes 2024-09-17T23:08:04.477 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:14300 IOs, 5950261248 bytes 2024-09-17T23:08:04.539 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:14400 IOs, 5967831040 bytes 2024-09-17T23:08:04.627 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:14500 IOs, 6015642624 bytes 2024-09-17T23:08:04.704 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ error_code=124 2024-09-17T23:08:04.704 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set -e 2024-09-17T23:08:04.704 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' 124 -eq 124 ']' 2024-09-17T23:08:04.704 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:08:04.705 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-17T23:08:04.705 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:08:04.705 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:08:04.705 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:08:04.705 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state_pattern=up+replaying 2024-09-17T23:08:04.705 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local description_pattern= 2024-09-17T23:08:04.705 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local service_pattern= 2024-09-17T23:08:04.705 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:08:04.705 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:08:05.706 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-17T23:08:05.706 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:08:05.706 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:08:05.706 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:08:05.706 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state_pattern=up+replaying 2024-09-17T23:08:05.707 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local description_pattern= 2024-09-17T23:08:05.707 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local service_pattern= 2024-09-17T23:08:05.707 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status 2024-09-17T23:08:05.707 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ CEPH_ARGS= 2024-09-17T23:08:05.707 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-17T23:08:05.965 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ status='test: 2024-09-17T23:08:05.965 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: global_id: d6dfe272-2e04-4592-a96a-96648936ad18 2024-09-17T23:08:05.965 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: state: up+replaying 2024-09-17T23:08:05.965 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: description: replaying, {"bytes_per_second":35724288.0,"bytes_per_snapshot":535864320.0,"last_snapshot_bytes":535864320,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726614464,"remote_snapshot_timestamp":1726614464,"replay_state":"idle"} 2024-09-17T23:08:05.965 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: service: mirror.3 on smithi038 2024-09-17T23:08:05.965 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: last_update: 2024-09-17 23:07:55 2024-09-17T23:08:05.965 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: peer_sites: 2024-09-17T23:08:05.965 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: name: cluster2 2024-09-17T23:08:05.965 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: state: up+stopped 2024-09-17T23:08:05.966 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: description: local image is primary 2024-09-17T23:08:05.966 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: last_update: 2024-09-17 23:07:54' 2024-09-17T23:08:05.966 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep '^ state: .*up+replaying' 2024-09-17T23:08:05.966 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: state: up+replaying 2024-09-17T23:08:05.966 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep '^ description: .*' 2024-09-17T23:08:05.967 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: description: replaying, {"bytes_per_second":35724288.0,"bytes_per_snapshot":535864320.0,"last_snapshot_bytes":535864320,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726614464,"remote_snapshot_timestamp":1726614464,"replay_state":"idle"} 2024-09-17T23:08:05.967 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n '' ']' 2024-09-17T23:08:05.968 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ echo up+replaying 2024-09-17T23:08:05.968 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep '^up+' 2024-09-17T23:08:05.969 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:up+replaying 2024-09-17T23:08:05.969 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:08:05.970 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: service: mirror.3 on smithi038 2024-09-17T23:08:05.971 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local last_update 2024-09-17T23:08:05.971 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:08:05.973 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ last_update='2024-09-17 23:07:55' 2024-09-17T23:08:05.973 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-17 23:07:55' 2024-09-17T23:08:05.974 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:08:05.974 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:08:05.974 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:08:05.974 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state_pattern=up+replaying 2024-09-17T23:08:05.974 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local 'prev_last_update=2024-09-17 23:07:55' 2024-09-17T23:08:05.974 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status 2024-09-17T23:08:05.974 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ CEPH_ARGS= 2024-09-17T23:08:05.974 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-17T23:08:06.035 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ status='OKOKOK142214298mirror.2smithi03819.1.1-330-gc2ddf972falseOK42234294mirror.3smithi03819.1.1-330-gc2ddf972trueOK42624310mirror.1smithi03819.1.1-330-gc2ddf972falseOK43374340mirror.0smithi03819.1.1-330-gc2ddf972falseOKtestd6dfe272-2e04-4592-a96a-96648936ad18up+replayingreplaying, {"bytes_per_second":35724288.0,"bytes_per_snapshot":535864320.0,"last_snapshot_bytes":535864320,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726614464,"remote_snapshot_timestamp":1726614464,"replay_state":"idle"}42234294mirror.3smithi0382024-09-17 23:07:55cluster20ba5654d-47fb-4672-b20c-7359a1340314up+stoppedlocal image is primary2024-09-17 23:07:54' 2024-09-17T23:08:06.035 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local last_update state 2024-09-17T23:08:06.035 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-17T23:08:06.037 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ last_update='2024-09-17 23:07:55' 2024-09-17T23:08:06.038 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-17T23:08:06.040 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ state=up+replaying 2024-09-17T23:08:06.040 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ echo up+replaying 2024-09-17T23:08:06.040 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep up+replaying 2024-09-17T23:08:06.041 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:up+replaying 2024-09-17T23:08:06.042 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:08:06.042 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:08:06.042 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ snap_name=snap2 2024-09-17T23:08:06.042 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_snap cluster2 mirror test snap2 2024-09-17T23:08:06.042 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:08:06.042 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:08:06.042 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:08:06.042 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap2 2024-09-17T23:08:06.042 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap2 --debug-rbd=20 --debug-journaler=20 2024-09-17T23:08:33.779 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-17T23:08:33.780 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:08:33.780 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:08:33.780 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:08:33.780 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-17T23:08:33.780 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:08:33.780 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:08:33.780 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:08:33.780 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=started 2024-09-17T23:08:33.780 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:08:33.780 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:08:33.780 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:08:34.780 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-17T23:08:34.780 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:08:34.781 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:08:34.781 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:08:34.781 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=started 2024-09-17T23:08:34.781 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_result 2024-09-17T23:08:34.781 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=stopped 2024-09-17T23:08:34.782 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-17T23:08:34.782 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ grep -i state 2024-09-17T23:08:34.782 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_instance=cluster1 2024-09-17T23:08:34.782 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:08:34.782 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:08:34.782 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance=cluster1 2024-09-17T23:08:34.782 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local loop_instance 2024-09-17T23:08:34.782 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:08:34.782 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sleep 0 2024-09-17T23:08:34.783 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-17T23:08:34.783 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ seq 0 3 2024-09-17T23:08:34.785 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:08:34.785 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-17T23:08:34.785 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:0 2024-09-17T23:08:34.785 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:08:34.785 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:08:34.785 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-17T23:08:34.785 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:0 2024-09-17T23:08:34.785 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:08:34.785 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:08:34.785 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:08:34.785 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=0 2024-09-17T23:08:34.786 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-17T23:08:34.786 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:08:34.786 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:08:34.786 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=0 2024-09-17T23:08:34.786 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:08:34.787 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-17T23:08:34.787 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:0 2024-09-17T23:08:34.787 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:08:34.787 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:08:34.787 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-17T23:08:34.788 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:0 2024-09-17T23:08:34.788 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:08:34.788 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:08:34.788 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:08:34.788 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:08:34.788 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-17T23:08:34.788 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:08:34.788 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:08:34.788 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=0 2024-09-17T23:08:34.788 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=0 2024-09-17T23:08:34.788 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-17T23:08:34.808 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:08:34.809 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:08:34.809 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:08:34.809 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-17T23:08:34.892 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-17T23:08:34.892 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd mirror status : get status for rbd mirror 2024-09-17T23:08:34.892 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:admin_socket: invalid command 2024-09-17T23:08:34.896 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:08:34.896 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-17T23:08:34.896 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:1 2024-09-17T23:08:34.896 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:08:34.896 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:08:34.896 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-17T23:08:34.896 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:1 2024-09-17T23:08:34.896 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:08:34.896 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:08:34.896 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:08:34.897 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=1 2024-09-17T23:08:34.897 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-17T23:08:34.897 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:08:34.897 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:08:34.897 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=1 2024-09-17T23:08:34.897 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=1 2024-09-17T23:08:34.897 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-17T23:08:34.897 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:1 2024-09-17T23:08:34.897 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:08:34.897 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:08:34.897 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-17T23:08:34.897 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:1 2024-09-17T23:08:34.897 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:08:34.898 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:08:34.898 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:08:34.898 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=1 2024-09-17T23:08:34.898 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-17T23:08:34.898 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:08:34.898 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:08:34.898 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=1 2024-09-17T23:08:34.898 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=1 2024-09-17T23:08:34.898 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-17T23:08:34.915 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-17T23:08:34.915 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-17T23:08:34.915 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-17T23:08:34.915 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-17T23:08:34.993 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-17T23:08:34.993 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd mirror status : get status for rbd mirror 2024-09-17T23:08:34.993 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:admin_socket: invalid command 2024-09-17T23:08:34.996 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:08:34.996 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-17T23:08:34.996 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:2 2024-09-17T23:08:34.997 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:08:34.997 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:08:34.997 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-17T23:08:34.997 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:2 2024-09-17T23:08:34.997 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:08:34.997 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:08:34.997 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:08:34.997 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=2 2024-09-17T23:08:34.997 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-17T23:08:34.997 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:08:34.997 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:08:34.998 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=2 2024-09-17T23:08:34.998 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=2 2024-09-17T23:08:34.998 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-17T23:08:34.998 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:2 2024-09-17T23:08:34.998 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:08:34.998 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:08:34.998 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-17T23:08:34.998 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:2 2024-09-17T23:08:34.998 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:08:34.998 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:08:34.998 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:08:34.998 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=2 2024-09-17T23:08:34.998 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-17T23:08:34.998 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:08:34.998 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:08:34.999 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=2 2024-09-17T23:08:34.999 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=2 2024-09-17T23:08:34.999 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-17T23:08:35.015 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-17T23:08:35.015 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-17T23:08:35.015 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-17T23:08:35.015 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-17T23:08:35.091 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-17T23:08:35.091 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd mirror status : get status for rbd mirror 2024-09-17T23:08:35.091 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:admin_socket: invalid command 2024-09-17T23:08:35.094 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:08:35.094 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-09-17T23:08:35.094 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:3 2024-09-17T23:08:35.094 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:08:35.095 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:08:35.095 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-17T23:08:35.095 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:3 2024-09-17T23:08:35.095 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:08:35.095 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:08:35.095 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:08:35.095 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=3 2024-09-17T23:08:35.095 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-17T23:08:35.095 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:08:35.095 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:08:35.095 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=3 2024-09-17T23:08:35.095 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=3 2024-09-17T23:08:35.095 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-17T23:08:35.096 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:3 2024-09-17T23:08:35.096 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:08:35.096 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:08:35.096 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-17T23:08:35.096 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:3 2024-09-17T23:08:35.096 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:08:35.096 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:08:35.096 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:08:35.096 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=3 2024-09-17T23:08:35.096 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-17T23:08:35.096 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:08:35.096 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:08:35.096 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=3 2024-09-17T23:08:35.097 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=3 2024-09-17T23:08:35.097 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-17T23:08:35.112 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-17T23:08:35.112 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-17T23:08:35.112 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-17T23:08:35.112 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-09-17T23:08:35.197 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ return 0 2024-09-17T23:08:35.197 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ status_result=' "state": "Replaying"' 2024-09-17T23:08:35.197 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ echo ' "state": "Replaying"' 2024-09-17T23:08:35.197 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep -i Replaying 2024-09-17T23:08:35.198 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: "state": "Replaying" 2024-09-17T23:08:35.200 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_state=started 2024-09-17T23:08:35.200 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test started = started 2024-09-17T23:08:35.200 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:08:35.200 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-17T23:08:35.200 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:08:35.200 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:08:35.200 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:08:35.200 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:08:35.200 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = journal ']' 2024-09-17T23:08:35.200 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = snapshot ']' 2024-09-17T23:08:35.200 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-09-17T23:08:35.200 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:08:35.200 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:08:35.201 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:08:35.201 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:08:35.201 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster2-mirror-test.status 2024-09-17T23:08:35.201 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster2-mirror-test.status 2024-09-17T23:08:35.201 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:08:35.202 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-test.status 2024-09-17T23:08:35.202 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster1-mirror-test.status 2024-09-17T23:08:35.202 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster1-mirror-test.status 2024-09-17T23:08:35.202 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:08:35.203 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_status_log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-test.status 2024-09-17T23:08:35.203 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-17T23:08:35.203 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:08:35.203 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:08:35.203 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:08:35.203 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-17T23:08:36.771 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Snapshot ID: 8 2024-09-17T23:08:36.791 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.status 2024-09-17T23:08:36.791 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:08:36.791 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:08:36.792 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:08:36.792 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-test.status 2024-09-17T23:08:36.792 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-17T23:08:36.792 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:08:36.841 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-17T23:08:36.843 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snapshot_id=8 2024-09-17T23:08:36.843 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:08:36.843 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:08:36.843 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.2 2024-09-17T23:08:37.045 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.status 2024-09-17T23:08:37.045 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:08:37.045 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:08:37.045 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:08:37.045 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-test.status 2024-09-17T23:08:37.045 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-17T23:08:37.045 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:08:37.099 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:08:37.099 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:08:37.101 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:08:37.101 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:08:37.101 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:08:37.101 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:08:37.101 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 8 = '' 2024-09-17T23:08:37.101 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:08:37.102 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.4 2024-09-17T23:08:37.505 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.status 2024-09-17T23:08:37.505 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:08:37.505 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:08:37.505 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:08:37.505 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-test.status 2024-09-17T23:08:37.505 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-17T23:08:37.505 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:08:37.558 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:08:37.558 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:08:37.560 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:08:37.560 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:08:37.560 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:08:37.560 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:08:37.560 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 8 = '' 2024-09-17T23:08:37.561 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:08:37.561 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.8 2024-09-17T23:08:38.362 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.status 2024-09-17T23:08:38.362 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:08:38.363 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:08:38.363 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:08:38.363 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-test.status 2024-09-17T23:08:38.363 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-17T23:08:38.363 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:08:38.429 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:08:38.430 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:08:38.431 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:08:38.431 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:08:38.431 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:08:38.432 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:08:38.432 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 8 = '' 2024-09-17T23:08:38.432 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:08:38.432 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1.6 2024-09-17T23:08:40.033 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.status 2024-09-17T23:08:40.033 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:08:40.033 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:08:40.033 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:08:40.034 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-test.status 2024-09-17T23:08:40.034 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-17T23:08:40.034 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:08:40.085 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:08:40.088 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id=8 2024-09-17T23:08:40.088 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 8 = 8 2024-09-17T23:08:40.088 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:08:40.088 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snap_present cluster1 mirror test snap2 2024-09-17T23:08:40.088 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:08:40.088 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:08:40.088 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:08:40.088 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap2 2024-09-17T23:08:40.088 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:08:40.088 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-17T23:08:40.088 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:08:41.089 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 info mirror/test@snap2 2024-09-17T23:08:41.138 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'test': 2024-09-17T23:08:41.138 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 512 MiB in 128 objects 2024-09-17T23:08:41.138 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:08:41.138 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 4 2024-09-17T23:08:41.138 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 10c7e03f2edd 2024-09-17T23:08:41.138 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.10c7e03f2edd 2024-09-17T23:08:41.138 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:08:41.138 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-17T23:08:41.138 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:08:41.138 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:08:41.138 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:06:57 2024 2024-09-17T23:08:41.139 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:06:57 2024 2024-09-17T23:08:41.139 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:06:57 2024 2024-09-17T23:08:41.139 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: protected: False 2024-09-17T23:08:41.139 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:08:41.139 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:08:41.139 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: d6dfe272-2e04-4592-a96a-96648936ad18 2024-09-17T23:08:41.139 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: false 2024-09-17T23:08:41.142 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:08:41.142 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n snap1-clean ']' 2024-09-17T23:08:41.142 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ compare_image_snaps mirror test snap1-clean 2024-09-17T23:08:41.142 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:08:41.142 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:08:41.142 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap1-clean 2024-09-17T23:08:41.142 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local ret=0 2024-09-17T23:08:41.142 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local rmt_export=/tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export 2024-09-17T23:08:41.142 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local loc_export=/tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:08:41.142 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:08:41.143 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1-clean /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export 2024-09-17T23:08:41.653 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-17T23:08:41.662 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1-clean /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:08:42.410 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-17T23:08:42.419 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ cmp /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:08:42.665 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:08:42.784 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:08:42.784 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ compare_image_snaps mirror test snap2 2024-09-17T23:08:42.785 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:08:42.785 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:08:42.785 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap2 2024-09-17T23:08:42.785 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local ret=0 2024-09-17T23:08:42.785 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local rmt_export=/tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export 2024-09-17T23:08:42.785 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local loc_export=/tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:08:42.785 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:08:42.785 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2 /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export 2024-09-17T23:08:43.305 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-17T23:08:43.317 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2 /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:08:44.042 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-17T23:08:44.051 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ cmp /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:08:44.297 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:08:44.417 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:08:44.417 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ clean_snap_name=snap2-clean 2024-09-17T23:08:44.418 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_snap cluster2 mirror test snap2-clean 2024-09-17T23:08:44.418 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:08:44.418 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:08:44.418 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:08:44.418 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap2-clean 2024-09-17T23:08:44.418 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap2-clean --debug-rbd=20 --debug-journaler=20 2024-09-17T23:08:44.795 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 10` 2024-09-17T23:08:44.795 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ stress_write_image cluster2 mirror test 2024-09-17T23:08:44.795 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:08:44.795 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:08:44.795 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:08:44.795 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-17T23:08:44.796 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local duration=12 2024-09-17T23:08:44.797 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set +e 2024-09-17T23:08:44.797 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-17T23:08:45.207 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:100 IOs, 36474880 bytes 2024-09-17T23:08:45.320 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:200 IOs, 50537472 bytes 2024-09-17T23:08:45.460 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:300 IOs, 114741248 bytes 2024-09-17T23:08:45.589 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:400 IOs, 198601728 bytes 2024-09-17T23:08:45.650 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:500 IOs, 199212032 bytes 2024-09-17T23:08:45.750 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:600 IOs, 251445248 bytes 2024-09-17T23:08:45.855 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:700 IOs, 322030592 bytes 2024-09-17T23:08:45.957 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:800 IOs, 369935360 bytes 2024-09-17T23:08:46.059 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:900 IOs, 430602240 bytes 2024-09-17T23:08:46.212 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1000 IOs, 546446336 bytes 2024-09-17T23:08:46.332 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1100 IOs, 625830912 bytes 2024-09-17T23:08:46.445 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1200 IOs, 689709056 bytes 2024-09-17T23:08:46.503 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1300 IOs, 697964544 bytes 2024-09-17T23:08:46.574 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1400 IOs, 722750464 bytes 2024-09-17T23:08:46.626 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1500 IOs, 724629504 bytes 2024-09-17T23:08:46.721 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1600 IOs, 778687488 bytes 2024-09-17T23:08:46.800 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1700 IOs, 808187904 bytes 2024-09-17T23:08:46.868 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1800 IOs, 832833536 bytes 2024-09-17T23:08:46.933 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1900 IOs, 847855616 bytes 2024-09-17T23:08:47.001 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2000 IOs, 855797760 bytes 2024-09-17T23:08:47.247 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2100 IOs, 889371648 bytes 2024-09-17T23:08:47.332 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2200 IOs, 943187968 bytes 2024-09-17T23:08:47.441 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2300 IOs, 1004752896 bytes 2024-09-17T23:08:47.536 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2400 IOs, 1049115648 bytes 2024-09-17T23:08:47.640 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2500 IOs, 1108589568 bytes 2024-09-17T23:08:47.711 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2600 IOs, 1134049280 bytes 2024-09-17T23:08:47.825 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2700 IOs, 1210572800 bytes 2024-09-17T23:08:47.906 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2800 IOs, 1246958592 bytes 2024-09-17T23:08:47.975 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2900 IOs, 1267983360 bytes 2024-09-17T23:08:48.055 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3000 IOs, 1299796992 bytes 2024-09-17T23:08:48.147 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3100 IOs, 1347265536 bytes 2024-09-17T23:08:48.204 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3200 IOs, 1354346496 bytes 2024-09-17T23:08:48.332 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3300 IOs, 1438524416 bytes 2024-09-17T23:08:48.394 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3400 IOs, 1457349632 bytes 2024-09-17T23:08:48.510 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3500 IOs, 1527914496 bytes 2024-09-17T23:08:48.601 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3600 IOs, 1572753408 bytes 2024-09-17T23:08:48.670 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3700 IOs, 1598044160 bytes 2024-09-17T23:08:48.788 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3800 IOs, 1668675584 bytes 2024-09-17T23:08:48.905 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3900 IOs, 1742737408 bytes 2024-09-17T23:08:48.992 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4000 IOs, 1791476736 bytes 2024-09-17T23:08:49.087 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4100 IOs, 1836290048 bytes 2024-09-17T23:08:49.192 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4200 IOs, 1898811392 bytes 2024-09-17T23:08:49.269 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4300 IOs, 1928235008 bytes 2024-09-17T23:08:49.321 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4400 IOs, 1928825856 bytes 2024-09-17T23:08:49.468 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4500 IOs, 2034194432 bytes 2024-09-17T23:08:49.590 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4600 IOs, 2133242880 bytes 2024-09-17T23:08:49.723 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4700 IOs, 2201853952 bytes 2024-09-17T23:08:49.805 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4800 IOs, 2234387456 bytes 2024-09-17T23:08:49.884 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4900 IOs, 2271976448 bytes 2024-09-17T23:08:49.991 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5000 IOs, 2329906176 bytes 2024-09-17T23:08:50.077 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5100 IOs, 2369442816 bytes 2024-09-17T23:08:50.138 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5200 IOs, 2380470272 bytes 2024-09-17T23:08:50.227 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5300 IOs, 2421714944 bytes 2024-09-17T23:08:50.311 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5400 IOs, 2480904192 bytes 2024-09-17T23:08:50.445 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5500 IOs, 2566475776 bytes 2024-09-17T23:08:50.593 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5600 IOs, 2654363648 bytes 2024-09-17T23:08:50.646 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5700 IOs, 2655931392 bytes 2024-09-17T23:08:50.751 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5800 IOs, 2719634432 bytes 2024-09-17T23:08:50.803 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5900 IOs, 2720244736 bytes 2024-09-17T23:08:50.893 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6000 IOs, 2783613952 bytes 2024-09-17T23:08:51.010 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6100 IOs, 2840434688 bytes 2024-09-17T23:08:51.140 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6200 IOs, 2925224960 bytes 2024-09-17T23:08:51.231 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6300 IOs, 2973213696 bytes 2024-09-17T23:08:51.306 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6400 IOs, 2998949888 bytes 2024-09-17T23:08:51.366 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6500 IOs, 3008136192 bytes 2024-09-17T23:08:51.426 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6600 IOs, 3019064320 bytes 2024-09-17T23:08:51.557 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6700 IOs, 3107727360 bytes 2024-09-17T23:08:51.625 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6800 IOs, 3127446528 bytes 2024-09-17T23:08:51.709 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6900 IOs, 3171135488 bytes 2024-09-17T23:08:51.793 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7000 IOs, 3201388544 bytes 2024-09-17T23:08:51.868 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7100 IOs, 3228706816 bytes 2024-09-17T23:08:51.938 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7200 IOs, 3250237440 bytes 2024-09-17T23:08:52.020 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7300 IOs, 3286299648 bytes 2024-09-17T23:08:52.082 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7400 IOs, 3299512320 bytes 2024-09-17T23:08:52.145 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7500 IOs, 3315482624 bytes 2024-09-17T23:08:52.218 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7600 IOs, 3347017728 bytes 2024-09-17T23:08:52.292 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7700 IOs, 3367185408 bytes 2024-09-17T23:08:52.382 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7800 IOs, 3413017600 bytes 2024-09-17T23:08:52.441 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7900 IOs, 3425000448 bytes 2024-09-17T23:08:52.513 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8000 IOs, 3442104320 bytes 2024-09-17T23:08:52.617 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8100 IOs, 3498073088 bytes 2024-09-17T23:08:52.699 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8200 IOs, 3537118208 bytes 2024-09-17T23:08:52.794 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8300 IOs, 3583003648 bytes 2024-09-17T23:08:52.859 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8400 IOs, 3599268864 bytes 2024-09-17T23:08:52.973 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8500 IOs, 3666499584 bytes 2024-09-17T23:08:53.092 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8600 IOs, 3742177280 bytes 2024-09-17T23:08:53.147 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8700 IOs, 3742777344 bytes 2024-09-17T23:08:53.223 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8800 IOs, 3769828352 bytes 2024-09-17T23:08:53.274 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8900 IOs, 3770550272 bytes 2024-09-17T23:08:53.375 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9000 IOs, 3827656704 bytes 2024-09-17T23:08:53.490 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9100 IOs, 3896418304 bytes 2024-09-17T23:08:53.607 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9200 IOs, 3969133568 bytes 2024-09-17T23:08:53.670 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9300 IOs, 3982252032 bytes 2024-09-17T23:08:53.761 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9400 IOs, 4020697088 bytes 2024-09-17T23:08:53.891 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9500 IOs, 4105388032 bytes 2024-09-17T23:08:53.974 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9600 IOs, 4141210624 bytes 2024-09-17T23:08:54.053 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9700 IOs, 4166955008 bytes 2024-09-17T23:08:54.154 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9800 IOs, 4225128448 bytes 2024-09-17T23:08:54.206 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9900 IOs, 4225737728 bytes 2024-09-17T23:08:54.308 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10000 IOs, 4280953856 bytes 2024-09-17T23:08:54.382 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10100 IOs, 4305918976 bytes 2024-09-17T23:08:54.435 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10200 IOs, 4308423680 bytes 2024-09-17T23:08:54.544 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10300 IOs, 4371976192 bytes 2024-09-17T23:08:54.652 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10400 IOs, 4429157376 bytes 2024-09-17T23:08:54.748 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10500 IOs, 4475934720 bytes 2024-09-17T23:08:54.864 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10600 IOs, 4547451904 bytes 2024-09-17T23:08:54.923 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10700 IOs, 4551668736 bytes 2024-09-17T23:08:54.987 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10800 IOs, 4563693568 bytes 2024-09-17T23:08:55.110 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10900 IOs, 4639107072 bytes 2024-09-17T23:08:55.193 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:11000 IOs, 4673815552 bytes 2024-09-17T23:08:55.292 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:11100 IOs, 4738671616 bytes 2024-09-17T23:08:55.407 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:11200 IOs, 4784595968 bytes 2024-09-17T23:08:55.547 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:11300 IOs, 4879516672 bytes 2024-09-17T23:08:55.608 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:11400 IOs, 4887366656 bytes 2024-09-17T23:08:55.756 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:11500 IOs, 4991101952 bytes 2024-09-17T23:08:55.846 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:11600 IOs, 5031526400 bytes 2024-09-17T23:08:55.927 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:11700 IOs, 5063686144 bytes 2024-09-17T23:08:56.045 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:11800 IOs, 5133972480 bytes 2024-09-17T23:08:56.148 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:11900 IOs, 5195365376 bytes 2024-09-17T23:08:56.239 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:12000 IOs, 5234766848 bytes 2024-09-17T23:08:56.351 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:12100 IOs, 5287775232 bytes 2024-09-17T23:08:56.420 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:12200 IOs, 5306489856 bytes 2024-09-17T23:08:56.473 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:12300 IOs, 5307097088 bytes 2024-09-17T23:08:56.552 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:12400 IOs, 5335755776 bytes 2024-09-17T23:08:56.642 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:12500 IOs, 5373659136 bytes 2024-09-17T23:08:56.733 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:12600 IOs, 5419457536 bytes 2024-09-17T23:08:56.805 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ error_code=124 2024-09-17T23:08:56.805 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set -e 2024-09-17T23:08:56.805 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' 124 -eq 124 ']' 2024-09-17T23:08:56.805 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:08:56.805 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-17T23:08:56.805 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:08:56.805 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:08:56.806 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:08:56.806 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state_pattern=up+replaying 2024-09-17T23:08:56.806 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local description_pattern= 2024-09-17T23:08:56.806 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local service_pattern= 2024-09-17T23:08:56.806 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:08:56.806 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:08:57.807 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-17T23:08:57.807 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:08:57.807 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:08:57.807 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:08:57.807 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state_pattern=up+replaying 2024-09-17T23:08:57.807 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local description_pattern= 2024-09-17T23:08:57.807 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local service_pattern= 2024-09-17T23:08:57.807 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status 2024-09-17T23:08:57.807 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ CEPH_ARGS= 2024-09-17T23:08:57.808 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-17T23:08:57.863 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ status='test: 2024-09-17T23:08:57.863 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: global_id: d6dfe272-2e04-4592-a96a-96648936ad18 2024-09-17T23:08:57.863 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: state: up+replaying 2024-09-17T23:08:57.863 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: description: replaying, {"bytes_per_second":17860505.6,"bytes_per_snapshot":535839744.0,"last_snapshot_bytes":535815168,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726614515,"remote_snapshot_timestamp":1726614515,"replay_state":"idle"} 2024-09-17T23:08:57.863 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: service: mirror.3 on smithi038 2024-09-17T23:08:57.863 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: last_update: 2024-09-17 23:08:55 2024-09-17T23:08:57.863 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: peer_sites: 2024-09-17T23:08:57.863 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: name: cluster2 2024-09-17T23:08:57.863 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: state: up+stopped 2024-09-17T23:08:57.864 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: description: local image is primary 2024-09-17T23:08:57.864 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: last_update: 2024-09-17 23:08:54' 2024-09-17T23:08:57.864 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep '^ state: .*up+replaying' 2024-09-17T23:08:57.864 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: state: up+replaying 2024-09-17T23:08:57.864 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep '^ description: .*' 2024-09-17T23:08:57.865 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: description: replaying, {"bytes_per_second":17860505.6,"bytes_per_snapshot":535839744.0,"last_snapshot_bytes":535815168,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726614515,"remote_snapshot_timestamp":1726614515,"replay_state":"idle"} 2024-09-17T23:08:57.866 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n '' ']' 2024-09-17T23:08:57.866 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ echo up+replaying 2024-09-17T23:08:57.866 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep '^up+' 2024-09-17T23:08:57.867 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:up+replaying 2024-09-17T23:08:57.867 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:08:57.868 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: service: mirror.3 on smithi038 2024-09-17T23:08:57.868 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local last_update 2024-09-17T23:08:57.868 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:08:57.871 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ last_update='2024-09-17 23:08:55' 2024-09-17T23:08:57.871 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-17 23:08:55' 2024-09-17T23:08:57.871 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:08:57.871 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:08:57.871 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:08:57.871 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state_pattern=up+replaying 2024-09-17T23:08:57.871 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local 'prev_last_update=2024-09-17 23:08:55' 2024-09-17T23:08:57.871 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status 2024-09-17T23:08:57.871 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ CEPH_ARGS= 2024-09-17T23:08:57.871 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-17T23:08:57.930 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ status='OKOKOK142214298mirror.2smithi03819.1.1-330-gc2ddf972falseOK42234294mirror.3smithi03819.1.1-330-gc2ddf972trueOK42624310mirror.1smithi03819.1.1-330-gc2ddf972falseOK43374340mirror.0smithi03819.1.1-330-gc2ddf972falseOKtestd6dfe272-2e04-4592-a96a-96648936ad18up+replayingreplaying, {"bytes_per_second":17860505.6,"bytes_per_snapshot":535839744.0,"last_snapshot_bytes":535815168,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726614515,"remote_snapshot_timestamp":1726614515,"replay_state":"idle"}42234294mirror.3smithi0382024-09-17 23:08:55cluster20ba5654d-47fb-4672-b20c-7359a1340314up+stoppedlocal image is primary2024-09-17 23:08:54' 2024-09-17T23:08:57.930 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local last_update state 2024-09-17T23:08:57.930 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-17T23:08:57.932 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ last_update='2024-09-17 23:08:55' 2024-09-17T23:08:57.932 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-17T23:08:57.934 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ state=up+replaying 2024-09-17T23:08:57.934 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ echo up+replaying 2024-09-17T23:08:57.934 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep up+replaying 2024-09-17T23:08:57.935 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:up+replaying 2024-09-17T23:08:57.935 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:08:57.935 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:08:57.936 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ snap_name=snap3 2024-09-17T23:08:57.936 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_snap cluster2 mirror test snap3 2024-09-17T23:08:57.936 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:08:57.936 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:08:57.936 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:08:57.936 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap3 2024-09-17T23:08:57.936 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap3 --debug-rbd=20 --debug-journaler=20 2024-09-17T23:09:07.784 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.4114698536609468) > weight (0.25) 2024-09-17T23:09:07.784 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.7011054034993499) > weight (0.25) 2024-09-17T23:09:07.784 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2024-09-17T23:09:07.785 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2024-09-17T23:09:07.785 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2024-09-17T23:09:07.786 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi038.stderr:2024-09-17T23:09:07.784+0000 7fe8e0812640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 36720) UID: 1000 2024-09-17T23:09:10.785 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2024-09-17T23:09:10.786 INFO:teuthology.orchestra.run:waiting for 600 2024-09-17T23:09:10.786 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2024-09-17T23:09:10.786 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2024-09-17T23:09:10.786 DEBUG:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2024-09-17T23:09:10.790 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2024-09-17T23:09:10.791 INFO:tasks.rbd_mirror_thrash:waiting for 86 secs before thrashing 2024-09-17T23:09:24.787 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.8126251782877717) > weight (0.25) 2024-09-17T23:09:24.787 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.8750007031451609) > weight (0.25) 2024-09-17T23:09:24.787 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.864458965122292) > weight (0.25) 2024-09-17T23:09:24.787 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.7631415329656436) > weight (0.25) 2024-09-17T23:09:24.787 INFO:tasks.rbd_mirror_thrash:waiting for 117 secs before thrashing 2024-09-17T23:09:28.690 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-17T23:09:28.690 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:09:28.690 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:09:28.690 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:09:28.690 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-17T23:09:28.691 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:09:28.691 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:09:28.691 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:09:28.691 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=started 2024-09-17T23:09:28.691 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:09:28.691 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:09:28.691 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:09:29.692 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-17T23:09:29.692 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:09:29.692 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:09:29.692 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:09:29.692 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=started 2024-09-17T23:09:29.692 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_result 2024-09-17T23:09:29.692 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=stopped 2024-09-17T23:09:29.693 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-17T23:09:29.693 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_instance=cluster1 2024-09-17T23:09:29.693 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ grep -i state 2024-09-17T23:09:29.693 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:09:29.693 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:09:29.693 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance=cluster1 2024-09-17T23:09:29.693 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local loop_instance 2024-09-17T23:09:29.693 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:09:29.693 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sleep 0 2024-09-17T23:09:29.694 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-17T23:09:29.694 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ seq 0 3 2024-09-17T23:09:29.696 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:09:29.696 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-17T23:09:29.696 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:0 2024-09-17T23:09:29.696 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:09:29.696 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:09:29.696 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-17T23:09:29.696 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:0 2024-09-17T23:09:29.696 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:09:29.696 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:09:29.696 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:09:29.696 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=0 2024-09-17T23:09:29.696 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-17T23:09:29.697 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:09:29.697 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:09:29.697 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=0 2024-09-17T23:09:29.697 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:09:29.697 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-17T23:09:29.698 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:0 2024-09-17T23:09:29.699 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:09:29.699 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:09:29.699 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-17T23:09:29.699 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:0 2024-09-17T23:09:29.699 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:09:29.699 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:09:29.699 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:09:29.699 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:09:29.699 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-17T23:09:29.699 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:09:29.699 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:09:29.700 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=0 2024-09-17T23:09:29.700 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=0 2024-09-17T23:09:29.700 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-17T23:09:29.718 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:09:29.718 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:09:29.719 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:09:29.719 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-17T23:09:29.804 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-17T23:09:29.804 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd mirror status : get status for rbd mirror 2024-09-17T23:09:29.804 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:admin_socket: invalid command 2024-09-17T23:09:29.807 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:09:29.808 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-17T23:09:29.808 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:1 2024-09-17T23:09:29.808 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:09:29.808 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:09:29.808 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-17T23:09:29.808 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:1 2024-09-17T23:09:29.808 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:09:29.808 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:09:29.808 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:09:29.808 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=1 2024-09-17T23:09:29.808 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-17T23:09:29.808 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:09:29.808 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:09:29.809 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=1 2024-09-17T23:09:29.809 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=1 2024-09-17T23:09:29.809 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-17T23:09:29.809 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:1 2024-09-17T23:09:29.809 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:09:29.809 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:09:29.809 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-17T23:09:29.809 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:1 2024-09-17T23:09:29.809 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:09:29.809 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:09:29.809 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:09:29.810 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=1 2024-09-17T23:09:29.810 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-17T23:09:29.810 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:09:29.810 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:09:29.810 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=1 2024-09-17T23:09:29.810 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=1 2024-09-17T23:09:29.810 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-17T23:09:29.827 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-17T23:09:29.827 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-17T23:09:29.827 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-17T23:09:29.827 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-17T23:09:29.904 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-17T23:09:29.904 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd mirror status : get status for rbd mirror 2024-09-17T23:09:29.904 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:admin_socket: invalid command 2024-09-17T23:09:29.907 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:09:29.907 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-17T23:09:29.907 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:2 2024-09-17T23:09:29.907 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:09:29.907 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:09:29.907 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-17T23:09:29.908 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:2 2024-09-17T23:09:29.908 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:09:29.908 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:09:29.908 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:09:29.908 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=2 2024-09-17T23:09:29.908 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-17T23:09:29.908 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:09:29.908 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:09:29.908 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=2 2024-09-17T23:09:29.908 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=2 2024-09-17T23:09:29.908 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-17T23:09:29.908 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:2 2024-09-17T23:09:29.908 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:09:29.908 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:09:29.909 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-17T23:09:29.909 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:2 2024-09-17T23:09:29.909 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:09:29.909 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:09:29.909 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:09:29.909 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=2 2024-09-17T23:09:29.909 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-17T23:09:29.909 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:09:29.909 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:09:29.909 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=2 2024-09-17T23:09:29.909 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=2 2024-09-17T23:09:29.909 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-17T23:09:29.926 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-17T23:09:29.926 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-17T23:09:29.926 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-17T23:09:29.926 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-17T23:09:30.002 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-17T23:09:30.002 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd mirror status : get status for rbd mirror 2024-09-17T23:09:30.002 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:admin_socket: invalid command 2024-09-17T23:09:30.005 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:09:30.005 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-09-17T23:09:30.005 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:3 2024-09-17T23:09:30.005 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:09:30.005 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:09:30.005 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-17T23:09:30.006 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:3 2024-09-17T23:09:30.006 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:09:30.006 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:09:30.006 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:09:30.006 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=3 2024-09-17T23:09:30.006 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-17T23:09:30.006 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:09:30.006 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:09:30.006 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=3 2024-09-17T23:09:30.006 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=3 2024-09-17T23:09:30.006 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-17T23:09:30.006 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:3 2024-09-17T23:09:30.006 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:09:30.006 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:09:30.007 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-17T23:09:30.007 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:3 2024-09-17T23:09:30.007 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:09:30.007 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:09:30.007 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:09:30.007 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=3 2024-09-17T23:09:30.007 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-17T23:09:30.007 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:09:30.007 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:09:30.007 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=3 2024-09-17T23:09:30.007 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=3 2024-09-17T23:09:30.007 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-17T23:09:30.024 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-17T23:09:30.024 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-17T23:09:30.024 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-17T23:09:30.024 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-09-17T23:09:30.109 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ return 0 2024-09-17T23:09:30.110 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ status_result=' "state": "Replaying"' 2024-09-17T23:09:30.110 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ echo ' "state": "Replaying"' 2024-09-17T23:09:30.110 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep -i Replaying 2024-09-17T23:09:30.111 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: "state": "Replaying" 2024-09-17T23:09:30.113 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_state=started 2024-09-17T23:09:30.113 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test started = started 2024-09-17T23:09:30.113 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:09:30.113 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-17T23:09:30.113 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:09:30.113 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:09:30.113 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:09:30.113 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:09:30.113 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = journal ']' 2024-09-17T23:09:30.113 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = snapshot ']' 2024-09-17T23:09:30.113 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-09-17T23:09:30.113 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:09:30.113 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:09:30.113 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:09:30.114 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:09:30.114 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster2-mirror-test.status 2024-09-17T23:09:30.114 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster2-mirror-test.status 2024-09-17T23:09:30.114 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:09:30.114 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-test.status 2024-09-17T23:09:30.114 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster1-mirror-test.status 2024-09-17T23:09:30.114 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster1-mirror-test.status 2024-09-17T23:09:30.115 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:09:30.115 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_status_log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-test.status 2024-09-17T23:09:30.115 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-17T23:09:30.115 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:09:30.116 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:09:30.116 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:09:30.116 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-17T23:09:31.673 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Snapshot ID: 11 2024-09-17T23:09:31.693 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.status 2024-09-17T23:09:31.693 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:09:31.693 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:09:31.693 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:09:31.693 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-test.status 2024-09-17T23:09:31.695 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:09:31.695 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-17T23:09:31.765 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-17T23:09:31.767 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snapshot_id=11 2024-09-17T23:09:31.767 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:09:31.767 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:09:31.767 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.2 2024-09-17T23:09:31.968 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.status 2024-09-17T23:09:31.968 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:09:31.969 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:09:31.969 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:09:31.969 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-test.status 2024-09-17T23:09:31.969 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-17T23:09:31.969 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:09:32.018 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:09:32.018 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:09:32.019 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:09:32.019 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:09:32.020 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:09:32.020 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:09:32.020 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 11 = '' 2024-09-17T23:09:32.021 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:09:32.021 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.4 2024-09-17T23:09:32.421 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.status 2024-09-17T23:09:32.421 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:09:32.422 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:09:32.422 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:09:32.422 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-test.status 2024-09-17T23:09:32.422 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-17T23:09:32.422 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:09:32.475 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:09:32.475 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:09:32.476 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:09:32.476 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:09:32.476 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:09:32.477 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:09:32.477 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 11 = '' 2024-09-17T23:09:32.477 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:09:32.477 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.8 2024-09-17T23:09:33.278 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.status 2024-09-17T23:09:33.279 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:09:33.279 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:09:33.279 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:09:33.279 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-test.status 2024-09-17T23:09:33.279 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-17T23:09:33.279 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:09:33.359 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:09:33.360 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:09:33.361 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:09:33.361 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:09:33.361 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:09:33.362 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:09:33.362 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 11 = '' 2024-09-17T23:09:33.362 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:09:33.362 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1.6 2024-09-17T23:09:34.963 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.status 2024-09-17T23:09:34.963 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:09:34.963 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:09:34.964 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:09:34.964 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-test.status 2024-09-17T23:09:34.964 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-17T23:09:34.964 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:09:35.017 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:09:35.019 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id=11 2024-09-17T23:09:35.019 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 11 = 11 2024-09-17T23:09:35.019 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:09:35.020 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snap_present cluster1 mirror test snap3 2024-09-17T23:09:35.020 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:09:35.020 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:09:35.020 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:09:35.020 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap3 2024-09-17T23:09:35.020 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:09:35.020 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-17T23:09:35.020 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:09:36.020 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 info mirror/test@snap3 2024-09-17T23:09:36.068 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'test': 2024-09-17T23:09:36.068 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 512 MiB in 128 objects 2024-09-17T23:09:36.069 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:09:36.069 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 6 2024-09-17T23:09:36.069 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 10c7e03f2edd 2024-09-17T23:09:36.069 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.10c7e03f2edd 2024-09-17T23:09:36.069 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:09:36.069 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-17T23:09:36.069 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:09:36.069 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:09:36.069 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:06:57 2024 2024-09-17T23:09:36.069 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:08:41 2024 2024-09-17T23:09:36.069 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:06:57 2024 2024-09-17T23:09:36.069 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: protected: False 2024-09-17T23:09:36.070 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:09:36.070 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:09:36.070 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: d6dfe272-2e04-4592-a96a-96648936ad18 2024-09-17T23:09:36.070 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: false 2024-09-17T23:09:36.072 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:09:36.072 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n snap2-clean ']' 2024-09-17T23:09:36.072 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ compare_image_snaps mirror test snap2-clean 2024-09-17T23:09:36.072 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:09:36.072 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:09:36.072 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap2-clean 2024-09-17T23:09:36.073 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local ret=0 2024-09-17T23:09:36.073 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local rmt_export=/tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export 2024-09-17T23:09:36.073 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local loc_export=/tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:09:36.073 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:09:36.073 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2-clean /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export 2024-09-17T23:09:36.582 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-17T23:09:36.592 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2-clean /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:09:37.323 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-17T23:09:37.332 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ cmp /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:09:37.578 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:09:37.697 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:09:37.697 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ compare_image_snaps mirror test snap3 2024-09-17T23:09:37.698 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:09:37.698 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:09:37.698 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap3 2024-09-17T23:09:37.698 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local ret=0 2024-09-17T23:09:37.698 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local rmt_export=/tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export 2024-09-17T23:09:37.698 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local loc_export=/tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:09:37.698 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:09:37.698 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3 /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export 2024-09-17T23:09:38.215 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-17T23:09:38.225 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3 /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:09:38.880 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-17T23:09:38.889 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ cmp /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:09:39.135 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:09:39.254 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:09:39.254 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ clean_snap_name=snap3-clean 2024-09-17T23:09:39.254 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_snap cluster2 mirror test snap3-clean 2024-09-17T23:09:39.254 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:09:39.254 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:09:39.254 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:09:39.254 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap3-clean 2024-09-17T23:09:39.254 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap3-clean --debug-rbd=20 --debug-journaler=20 2024-09-17T23:09:39.701 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 10` 2024-09-17T23:09:39.701 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ stress_write_image cluster2 mirror test 2024-09-17T23:09:39.701 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:09:39.701 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:09:39.701 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:09:39.701 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-17T23:09:39.703 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local duration=8 2024-09-17T23:09:39.703 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set +e 2024-09-17T23:09:39.703 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-17T23:09:39.983 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:100 IOs, 54242304 bytes 2024-09-17T23:09:40.101 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:200 IOs, 83065856 bytes 2024-09-17T23:09:40.195 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:300 IOs, 95344640 bytes 2024-09-17T23:09:40.302 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:400 IOs, 139387904 bytes 2024-09-17T23:09:40.359 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:500 IOs, 140010496 bytes 2024-09-17T23:09:40.443 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:600 IOs, 173075456 bytes 2024-09-17T23:09:40.601 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:700 IOs, 291007488 bytes 2024-09-17T23:09:40.687 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:800 IOs, 321351680 bytes 2024-09-17T23:09:40.772 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:900 IOs, 358955008 bytes 2024-09-17T23:09:40.847 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1000 IOs, 384003072 bytes 2024-09-17T23:09:40.945 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1100 IOs, 448818176 bytes 2024-09-17T23:09:41.148 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1200 IOs, 598649856 bytes 2024-09-17T23:09:41.209 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1300 IOs, 609091584 bytes 2024-09-17T23:09:41.320 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1400 IOs, 670795776 bytes 2024-09-17T23:09:41.382 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1500 IOs, 678366208 bytes 2024-09-17T23:09:41.436 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1600 IOs, 678963200 bytes 2024-09-17T23:09:41.548 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1700 IOs, 742843392 bytes 2024-09-17T23:09:41.635 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1800 IOs, 775149568 bytes 2024-09-17T23:09:41.739 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1900 IOs, 830932992 bytes 2024-09-17T23:09:41.863 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2000 IOs, 911937536 bytes 2024-09-17T23:09:42.000 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2100 IOs, 997401600 bytes 2024-09-17T23:09:42.092 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2200 IOs, 1052103680 bytes 2024-09-17T23:09:42.224 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2300 IOs, 1124833280 bytes 2024-09-17T23:09:42.286 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2400 IOs, 1137122304 bytes 2024-09-17T23:09:42.369 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2500 IOs, 1169716224 bytes 2024-09-17T23:09:42.461 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2600 IOs, 1211054080 bytes 2024-09-17T23:09:42.555 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2700 IOs, 1259657216 bytes 2024-09-17T23:09:42.620 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2800 IOs, 1266130944 bytes 2024-09-17T23:09:42.719 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2900 IOs, 1316221952 bytes 2024-09-17T23:09:42.826 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3000 IOs, 1367938048 bytes 2024-09-17T23:09:42.900 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3100 IOs, 1388690432 bytes 2024-09-17T23:09:42.977 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3200 IOs, 1416610816 bytes 2024-09-17T23:09:43.085 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3300 IOs, 1474370560 bytes 2024-09-17T23:09:43.174 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3400 IOs, 1512877056 bytes 2024-09-17T23:09:43.242 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3500 IOs, 1527410688 bytes 2024-09-17T23:09:43.311 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3600 IOs, 1542627328 bytes 2024-09-17T23:09:43.380 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3700 IOs, 1559109632 bytes 2024-09-17T23:09:43.434 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3800 IOs, 1559676928 bytes 2024-09-17T23:09:43.528 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3900 IOs, 1597963264 bytes 2024-09-17T23:09:43.638 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4000 IOs, 1658415104 bytes 2024-09-17T23:09:43.702 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4100 IOs, 1667798016 bytes 2024-09-17T23:09:43.781 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4200 IOs, 1698632704 bytes 2024-09-17T23:09:43.878 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4300 IOs, 1742280704 bytes 2024-09-17T23:09:44.005 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4400 IOs, 1822106624 bytes 2024-09-17T23:09:44.125 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4500 IOs, 1909882880 bytes 2024-09-17T23:09:44.234 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4600 IOs, 1957754880 bytes 2024-09-17T23:09:44.327 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4700 IOs, 1995230208 bytes 2024-09-17T23:09:44.401 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4800 IOs, 2017004544 bytes 2024-09-17T23:09:44.508 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4900 IOs, 2078075904 bytes 2024-09-17T23:09:44.594 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5000 IOs, 2116313088 bytes 2024-09-17T23:09:44.696 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5100 IOs, 2169608192 bytes 2024-09-17T23:09:44.814 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5200 IOs, 2230121472 bytes 2024-09-17T23:09:44.925 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5300 IOs, 2293003264 bytes 2024-09-17T23:09:45.007 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5400 IOs, 2321965056 bytes 2024-09-17T23:09:45.131 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5500 IOs, 2399001600 bytes 2024-09-17T23:09:45.231 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5600 IOs, 2445827072 bytes 2024-09-17T23:09:45.324 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5700 IOs, 2484840448 bytes 2024-09-17T23:09:45.388 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5800 IOs, 2493193216 bytes 2024-09-17T23:09:45.452 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5900 IOs, 2503343104 bytes 2024-09-17T23:09:45.576 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6000 IOs, 2586596352 bytes 2024-09-17T23:09:45.696 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6100 IOs, 2654283776 bytes 2024-09-17T23:09:45.828 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6200 IOs, 2735034368 bytes 2024-09-17T23:09:45.937 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6300 IOs, 2797650944 bytes 2024-09-17T23:09:46.095 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6400 IOs, 2928631808 bytes 2024-09-17T23:09:46.181 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6500 IOs, 2950337536 bytes 2024-09-17T23:09:46.275 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6600 IOs, 2995832832 bytes 2024-09-17T23:09:46.355 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6700 IOs, 3020841984 bytes 2024-09-17T23:09:46.485 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6800 IOs, 3106171904 bytes 2024-09-17T23:09:46.541 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6900 IOs, 3113460736 bytes 2024-09-17T23:09:46.658 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7000 IOs, 3176541184 bytes 2024-09-17T23:09:46.740 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7100 IOs, 3205228544 bytes 2024-09-17T23:09:46.856 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7200 IOs, 3225123840 bytes 2024-09-17T23:09:46.898 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7300 IOs, 3261223936 bytes 2024-09-17T23:09:47.021 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7400 IOs, 3332017152 bytes 2024-09-17T23:09:47.102 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7500 IOs, 3359478784 bytes 2024-09-17T23:09:47.211 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7600 IOs, 3416755200 bytes 2024-09-17T23:09:47.283 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7700 IOs, 3436255232 bytes 2024-09-17T23:09:47.365 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7800 IOs, 3466859520 bytes 2024-09-17T23:09:47.466 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7900 IOs, 3516060672 bytes 2024-09-17T23:09:47.572 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8000 IOs, 3572303872 bytes 2024-09-17T23:09:47.660 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8100 IOs, 3601429504 bytes 2024-09-17T23:09:47.710 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ error_code=124 2024-09-17T23:09:47.710 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set -e 2024-09-17T23:09:47.710 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' 124 -eq 124 ']' 2024-09-17T23:09:47.710 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:09:47.711 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-17T23:09:47.711 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:09:47.711 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:09:47.711 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:09:47.711 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state_pattern=up+replaying 2024-09-17T23:09:47.711 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local description_pattern= 2024-09-17T23:09:47.711 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local service_pattern= 2024-09-17T23:09:47.711 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:09:47.711 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:09:48.711 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-17T23:09:48.711 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:09:48.712 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:09:48.712 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:09:48.712 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state_pattern=up+replaying 2024-09-17T23:09:48.712 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local description_pattern= 2024-09-17T23:09:48.712 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local service_pattern= 2024-09-17T23:09:48.712 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status 2024-09-17T23:09:48.712 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ CEPH_ARGS= 2024-09-17T23:09:48.712 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-17T23:09:48.774 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ status='test: 2024-09-17T23:09:48.775 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: global_id: d6dfe272-2e04-4592-a96a-96648936ad18 2024-09-17T23:09:48.775 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: state: up+replaying 2024-09-17T23:09:48.775 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":535839744.0,"last_snapshot_bytes":535815168,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726614515,"remote_snapshot_timestamp":1726614515,"replay_state":"idle"} 2024-09-17T23:09:48.775 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: service: mirror.3 on smithi038 2024-09-17T23:09:48.775 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: last_update: 2024-09-17 23:09:25 2024-09-17T23:09:48.775 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: peer_sites: 2024-09-17T23:09:48.775 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: name: cluster2 2024-09-17T23:09:48.775 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: state: up+stopped 2024-09-17T23:09:48.775 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: description: local image is primary 2024-09-17T23:09:48.775 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: last_update: 2024-09-17 23:09:24' 2024-09-17T23:09:48.776 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep '^ state: .*up+replaying' 2024-09-17T23:09:48.776 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: state: up+replaying 2024-09-17T23:09:48.776 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep '^ description: .*' 2024-09-17T23:09:48.777 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":535839744.0,"last_snapshot_bytes":535815168,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726614515,"remote_snapshot_timestamp":1726614515,"replay_state":"idle"} 2024-09-17T23:09:48.777 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n '' ']' 2024-09-17T23:09:48.778 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ echo up+replaying 2024-09-17T23:09:48.778 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep '^up+' 2024-09-17T23:09:48.779 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:up+replaying 2024-09-17T23:09:48.779 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:09:48.780 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: service: mirror.3 on smithi038 2024-09-17T23:09:48.780 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local last_update 2024-09-17T23:09:48.780 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:09:48.782 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ last_update='2024-09-17 23:09:25' 2024-09-17T23:09:48.782 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-17 23:09:25' 2024-09-17T23:09:48.783 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:09:48.783 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:09:48.783 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:09:48.783 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state_pattern=up+replaying 2024-09-17T23:09:48.783 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local 'prev_last_update=2024-09-17 23:09:25' 2024-09-17T23:09:48.783 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status 2024-09-17T23:09:48.783 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ CEPH_ARGS= 2024-09-17T23:09:48.783 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-17T23:09:48.843 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ status='OKOKOK142214298mirror.2smithi03819.1.1-330-gc2ddf972falseOK42234294mirror.3smithi03819.1.1-330-gc2ddf972trueOK42624310mirror.1smithi03819.1.1-330-gc2ddf972falseOK43374340mirror.0smithi03819.1.1-330-gc2ddf972falseOKtestd6dfe272-2e04-4592-a96a-96648936ad18up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":535839744.0,"last_snapshot_bytes":535815168,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726614515,"remote_snapshot_timestamp":1726614515,"replay_state":"idle"}42234294mirror.3smithi0382024-09-17 23:09:25cluster20ba5654d-47fb-4672-b20c-7359a1340314up+stoppedlocal image is primary2024-09-17 23:09:24' 2024-09-17T23:09:48.843 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local last_update state 2024-09-17T23:09:48.843 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-17T23:09:48.845 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ last_update='2024-09-17 23:09:25' 2024-09-17T23:09:48.845 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-17T23:09:48.847 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ state=up+replaying 2024-09-17T23:09:48.847 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ echo up+replaying 2024-09-17T23:09:48.847 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep up+replaying 2024-09-17T23:09:48.848 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:up+replaying 2024-09-17T23:09:48.849 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:09:48.849 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:09:48.849 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ snap_name=snap4 2024-09-17T23:09:48.849 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_snap cluster2 mirror test snap4 2024-09-17T23:09:48.849 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:09:48.849 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:09:48.849 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:09:48.850 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap4 2024-09-17T23:09:48.850 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap4 --debug-rbd=20 --debug-journaler=20 2024-09-17T23:10:17.716 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-17T23:10:17.716 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:10:17.717 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:10:17.717 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:10:17.717 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-17T23:10:17.717 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:10:17.717 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:10:17.717 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:10:17.717 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=started 2024-09-17T23:10:17.717 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:10:17.717 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:10:17.717 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:10:18.717 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-17T23:10:18.717 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:10:18.717 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:10:18.717 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:10:18.718 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=started 2024-09-17T23:10:18.718 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_result 2024-09-17T23:10:18.718 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=stopped 2024-09-17T23:10:18.719 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-17T23:10:18.719 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_instance=cluster1 2024-09-17T23:10:18.719 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ grep -i state 2024-09-17T23:10:18.719 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:10:18.719 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:10:18.719 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance=cluster1 2024-09-17T23:10:18.719 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local loop_instance 2024-09-17T23:10:18.719 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:10:18.719 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sleep 0 2024-09-17T23:10:18.719 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-17T23:10:18.721 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ seq 0 3 2024-09-17T23:10:18.721 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:10:18.721 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-17T23:10:18.721 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:0 2024-09-17T23:10:18.721 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:10:18.721 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:10:18.721 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-17T23:10:18.721 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:0 2024-09-17T23:10:18.721 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:10:18.722 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:10:18.722 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:10:18.722 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=0 2024-09-17T23:10:18.722 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-17T23:10:18.722 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:10:18.722 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:10:18.722 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=0 2024-09-17T23:10:18.722 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:10:18.722 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-17T23:10:18.722 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:0 2024-09-17T23:10:18.723 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:10:18.723 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:10:18.723 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-17T23:10:18.724 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:0 2024-09-17T23:10:18.724 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:10:18.724 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:10:18.724 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:10:18.724 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:10:18.724 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-17T23:10:18.724 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:10:18.724 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:10:18.724 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=0 2024-09-17T23:10:18.724 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=0 2024-09-17T23:10:18.724 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-17T23:10:18.744 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:10:18.744 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:10:18.744 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:10:18.744 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-17T23:10:18.827 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-17T23:10:18.827 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd mirror status : get status for rbd mirror 2024-09-17T23:10:18.827 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:admin_socket: invalid command 2024-09-17T23:10:18.830 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:10:18.830 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-17T23:10:18.830 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:1 2024-09-17T23:10:18.830 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:10:18.830 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:10:18.830 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-17T23:10:18.830 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:1 2024-09-17T23:10:18.830 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:10:18.830 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:10:18.830 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:10:18.830 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=1 2024-09-17T23:10:18.831 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-17T23:10:18.831 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:10:18.831 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:10:18.831 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=1 2024-09-17T23:10:18.831 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=1 2024-09-17T23:10:18.831 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-17T23:10:18.831 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:1 2024-09-17T23:10:18.831 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:10:18.831 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:10:18.831 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-17T23:10:18.831 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:1 2024-09-17T23:10:18.831 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:10:18.831 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:10:18.831 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:10:18.832 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=1 2024-09-17T23:10:18.832 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-17T23:10:18.832 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:10:18.832 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:10:18.832 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=1 2024-09-17T23:10:18.832 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=1 2024-09-17T23:10:18.832 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-17T23:10:18.850 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-17T23:10:18.850 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-17T23:10:18.850 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-17T23:10:18.850 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-17T23:10:18.927 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-17T23:10:18.927 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd mirror status : get status for rbd mirror 2024-09-17T23:10:18.927 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:admin_socket: invalid command 2024-09-17T23:10:18.930 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:10:18.930 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-17T23:10:18.930 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:2 2024-09-17T23:10:18.930 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:10:18.930 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:10:18.930 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-17T23:10:18.930 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:2 2024-09-17T23:10:18.930 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:10:18.931 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:10:18.931 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:10:18.931 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=2 2024-09-17T23:10:18.931 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-17T23:10:18.931 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:10:18.931 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:10:18.931 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=2 2024-09-17T23:10:18.931 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=2 2024-09-17T23:10:18.931 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-17T23:10:18.931 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:2 2024-09-17T23:10:18.931 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:10:18.931 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:10:18.931 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-17T23:10:18.931 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:2 2024-09-17T23:10:18.931 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:10:18.932 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:10:18.932 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:10:18.932 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=2 2024-09-17T23:10:18.932 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-17T23:10:18.932 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:10:18.932 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:10:18.932 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=2 2024-09-17T23:10:18.932 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=2 2024-09-17T23:10:18.932 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-17T23:10:18.949 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-17T23:10:18.950 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-17T23:10:18.950 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-17T23:10:18.950 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-17T23:10:19.025 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-17T23:10:19.025 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd mirror status : get status for rbd mirror 2024-09-17T23:10:19.025 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:admin_socket: invalid command 2024-09-17T23:10:19.028 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:10:19.028 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-09-17T23:10:19.029 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:3 2024-09-17T23:10:19.029 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:10:19.029 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:10:19.029 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-17T23:10:19.029 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:3 2024-09-17T23:10:19.029 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:10:19.029 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:10:19.029 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:10:19.029 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=3 2024-09-17T23:10:19.029 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-17T23:10:19.029 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:10:19.029 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:10:19.029 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=3 2024-09-17T23:10:19.029 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=3 2024-09-17T23:10:19.029 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-17T23:10:19.030 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:3 2024-09-17T23:10:19.030 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:10:19.030 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:10:19.030 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-17T23:10:19.030 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:3 2024-09-17T23:10:19.030 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:10:19.030 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:10:19.030 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:10:19.030 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=3 2024-09-17T23:10:19.030 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-17T23:10:19.030 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:10:19.030 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:10:19.030 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=3 2024-09-17T23:10:19.030 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=3 2024-09-17T23:10:19.031 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-17T23:10:19.047 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-17T23:10:19.047 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-17T23:10:19.047 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-17T23:10:19.047 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-09-17T23:10:19.132 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ return 0 2024-09-17T23:10:19.133 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ status_result=' "state": "Replaying"' 2024-09-17T23:10:19.133 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ echo ' "state": "Replaying"' 2024-09-17T23:10:19.133 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep -i Replaying 2024-09-17T23:10:19.133 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: "state": "Replaying" 2024-09-17T23:10:19.135 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_state=started 2024-09-17T23:10:19.135 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test started = started 2024-09-17T23:10:19.135 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:10:19.135 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-17T23:10:19.135 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:10:19.135 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:10:19.135 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:10:19.136 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:10:19.136 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = journal ']' 2024-09-17T23:10:19.136 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = snapshot ']' 2024-09-17T23:10:19.136 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-09-17T23:10:19.136 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:10:19.136 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:10:19.136 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:10:19.136 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:10:19.136 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster2-mirror-test.status 2024-09-17T23:10:19.136 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster2-mirror-test.status 2024-09-17T23:10:19.136 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:10:19.136 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-test.status 2024-09-17T23:10:19.136 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster1-mirror-test.status 2024-09-17T23:10:19.136 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster1-mirror-test.status 2024-09-17T23:10:19.137 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:10:19.137 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_status_log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-test.status 2024-09-17T23:10:19.137 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-17T23:10:19.137 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:10:19.138 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:10:19.138 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:10:19.138 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-17T23:10:20.721 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Snapshot ID: 14 2024-09-17T23:10:20.742 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.status 2024-09-17T23:10:20.742 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:10:20.742 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:10:20.742 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:10:20.742 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-test.status 2024-09-17T23:10:20.742 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-17T23:10:20.742 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:10:20.814 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-17T23:10:20.816 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snapshot_id=14 2024-09-17T23:10:20.816 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:10:20.816 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:10:20.816 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.2 2024-09-17T23:10:21.017 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.status 2024-09-17T23:10:21.018 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:10:21.018 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:10:21.018 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:10:21.018 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-test.status 2024-09-17T23:10:21.018 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-17T23:10:21.018 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:10:21.070 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:10:21.070 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:10:21.071 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:10:21.071 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:10:21.072 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:10:21.072 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:10:21.072 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 14 = '' 2024-09-17T23:10:21.072 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:10:21.072 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.4 2024-09-17T23:10:21.474 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.status 2024-09-17T23:10:21.474 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:10:21.474 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:10:21.474 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:10:21.474 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-test.status 2024-09-17T23:10:21.474 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-17T23:10:21.474 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:10:21.529 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:10:21.529 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:10:21.530 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:10:21.530 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:10:21.530 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:10:21.531 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:10:21.531 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 14 = '' 2024-09-17T23:10:21.531 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:10:21.531 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.8 2024-09-17T23:10:22.332 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.status 2024-09-17T23:10:22.333 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:10:22.333 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:10:22.333 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:10:22.333 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-test.status 2024-09-17T23:10:22.333 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-17T23:10:22.333 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:10:22.387 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:10:22.387 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:10:22.389 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:10:22.389 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:10:22.389 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:10:22.389 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:10:22.389 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 14 = '' 2024-09-17T23:10:22.390 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:10:22.390 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1.6 2024-09-17T23:10:23.991 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.status 2024-09-17T23:10:23.991 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:10:23.991 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:10:23.991 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:10:23.991 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-test.status 2024-09-17T23:10:23.991 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-17T23:10:23.991 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:10:24.042 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:10:24.044 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id=14 2024-09-17T23:10:24.044 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 14 = 14 2024-09-17T23:10:24.044 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:10:24.044 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snap_present cluster1 mirror test snap4 2024-09-17T23:10:24.044 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:10:24.045 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:10:24.045 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:10:24.045 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap4 2024-09-17T23:10:24.045 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:10:24.045 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-17T23:10:24.045 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:10:25.045 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 info mirror/test@snap4 2024-09-17T23:10:25.094 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'test': 2024-09-17T23:10:25.094 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 512 MiB in 128 objects 2024-09-17T23:10:25.094 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:10:25.094 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 8 2024-09-17T23:10:25.095 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 10c7e03f2edd 2024-09-17T23:10:25.095 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.10c7e03f2edd 2024-09-17T23:10:25.095 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:10:25.095 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-17T23:10:25.095 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:10:25.095 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:10:25.095 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:06:57 2024 2024-09-17T23:10:25.095 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:08:41 2024 2024-09-17T23:10:25.095 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:06:57 2024 2024-09-17T23:10:25.095 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: protected: False 2024-09-17T23:10:25.095 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:10:25.095 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:10:25.095 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: d6dfe272-2e04-4592-a96a-96648936ad18 2024-09-17T23:10:25.096 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: false 2024-09-17T23:10:25.098 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:10:25.098 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n snap3-clean ']' 2024-09-17T23:10:25.098 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ compare_image_snaps mirror test snap3-clean 2024-09-17T23:10:25.098 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:10:25.098 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:10:25.098 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap3-clean 2024-09-17T23:10:25.099 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local ret=0 2024-09-17T23:10:25.099 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local rmt_export=/tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export 2024-09-17T23:10:25.099 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local loc_export=/tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:10:25.099 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:10:25.099 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3-clean /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export 2024-09-17T23:10:25.617 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-17T23:10:25.627 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3-clean /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:10:26.341 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-17T23:10:26.352 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ cmp /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:10:26.598 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:10:26.717 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:10:26.717 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ compare_image_snaps mirror test snap4 2024-09-17T23:10:26.717 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:10:26.717 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:10:26.717 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap4 2024-09-17T23:10:26.717 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local ret=0 2024-09-17T23:10:26.718 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local rmt_export=/tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export 2024-09-17T23:10:26.718 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local loc_export=/tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:10:26.718 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:10:26.718 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4 /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export 2024-09-17T23:10:27.252 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-17T23:10:27.262 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4 /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:10:27.942 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-17T23:10:27.951 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ cmp /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:10:28.195 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:10:28.314 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:10:28.314 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ clean_snap_name=snap4-clean 2024-09-17T23:10:28.314 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_snap cluster2 mirror test snap4-clean 2024-09-17T23:10:28.314 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:10:28.315 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:10:28.315 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:10:28.315 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap4-clean 2024-09-17T23:10:28.315 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap4-clean --debug-rbd=20 --debug-journaler=20 2024-09-17T23:10:29.019 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 10` 2024-09-17T23:10:29.019 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ stress_write_image cluster2 mirror test 2024-09-17T23:10:29.020 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:10:29.020 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:10:29.020 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:10:29.020 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-17T23:10:29.021 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local duration=7 2024-09-17T23:10:29.021 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set +e 2024-09-17T23:10:29.021 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-17T23:10:29.315 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:100 IOs, 68276224 bytes 2024-09-17T23:10:29.461 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:200 IOs, 110461952 bytes 2024-09-17T23:10:29.581 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:300 IOs, 167318528 bytes 2024-09-17T23:10:29.695 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:400 IOs, 219717632 bytes 2024-09-17T23:10:29.770 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:500 IOs, 235857920 bytes 2024-09-17T23:10:29.863 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:600 IOs, 283246592 bytes 2024-09-17T23:10:29.941 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:700 IOs, 294252544 bytes 2024-09-17T23:10:30.086 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:800 IOs, 388628480 bytes 2024-09-17T23:10:30.177 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:900 IOs, 431790080 bytes 2024-09-17T23:10:30.270 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1000 IOs, 464195584 bytes 2024-09-17T23:10:30.341 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1100 IOs, 480791552 bytes 2024-09-17T23:10:30.406 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1200 IOs, 491164672 bytes 2024-09-17T23:10:30.479 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1300 IOs, 514963456 bytes 2024-09-17T23:10:30.605 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1400 IOs, 589819904 bytes 2024-09-17T23:10:30.692 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1500 IOs, 631643136 bytes 2024-09-17T23:10:30.768 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1600 IOs, 645692416 bytes 2024-09-17T23:10:30.879 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1700 IOs, 703005696 bytes 2024-09-17T23:10:30.956 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1800 IOs, 722894848 bytes 2024-09-17T23:10:31.027 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1900 IOs, 737956864 bytes 2024-09-17T23:10:31.124 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2000 IOs, 780428288 bytes 2024-09-17T23:10:31.196 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2100 IOs, 797878272 bytes 2024-09-17T23:10:31.288 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2200 IOs, 837977088 bytes 2024-09-17T23:10:31.371 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2300 IOs, 868126720 bytes 2024-09-17T23:10:31.447 INFO:tasks.ceph.cluster2.osd.1.smithi038.stderr:2024-09-17T23:10:31.445+0000 7f054ba99640 -1 reset not still connected to 0x55df0c21ac30 2024-09-17T23:10:31.508 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2400 IOs, 953781248 bytes 2024-09-17T23:10:31.616 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2500 IOs, 1004891136 bytes 2024-09-17T23:10:31.693 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2600 IOs, 1032328192 bytes 2024-09-17T23:10:31.759 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2700 IOs, 1041226752 bytes 2024-09-17T23:10:31.880 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2800 IOs, 1107135488 bytes 2024-09-17T23:10:31.959 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2900 IOs, 1125230592 bytes 2024-09-17T23:10:32.039 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3000 IOs, 1150075904 bytes 2024-09-17T23:10:32.127 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3100 IOs, 1182912512 bytes 2024-09-17T23:10:32.202 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3200 IOs, 1204875264 bytes 2024-09-17T23:10:32.294 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3300 IOs, 1242539008 bytes 2024-09-17T23:10:32.375 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3400 IOs, 1268191232 bytes 2024-09-17T23:10:32.445 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3500 IOs, 1280891904 bytes 2024-09-17T23:10:32.560 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3600 IOs, 1351242752 bytes 2024-09-17T23:10:32.671 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3700 IOs, 1410079744 bytes 2024-09-17T23:10:32.784 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3800 IOs, 1488040960 bytes 2024-09-17T23:10:32.879 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3900 IOs, 1509627904 bytes 2024-09-17T23:10:32.985 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4000 IOs, 1560008704 bytes 2024-09-17T23:10:33.069 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4100 IOs, 1587718144 bytes 2024-09-17T23:10:33.150 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4200 IOs, 1615000576 bytes 2024-09-17T23:10:33.212 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4300 IOs, 1620853760 bytes 2024-09-17T23:10:33.291 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4400 IOs, 1644393472 bytes 2024-09-17T23:10:33.368 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4500 IOs, 1664921600 bytes 2024-09-17T23:10:33.460 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4600 IOs, 1706817536 bytes 2024-09-17T23:10:33.523 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4700 IOs, 1711099904 bytes 2024-09-17T23:10:33.636 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4800 IOs, 1777928192 bytes 2024-09-17T23:10:33.737 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4900 IOs, 1811501056 bytes 2024-09-17T23:10:33.848 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5000 IOs, 1865145344 bytes 2024-09-17T23:10:33.916 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5100 IOs, 1877160960 bytes 2024-09-17T23:10:34.040 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5200 IOs, 1952369664 bytes 2024-09-17T23:10:34.158 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5300 IOs, 2016030720 bytes 2024-09-17T23:10:34.232 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5400 IOs, 2036143104 bytes 2024-09-17T23:10:34.372 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5500 IOs, 2122700800 bytes 2024-09-17T23:10:34.475 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5600 IOs, 2175213568 bytes 2024-09-17T23:10:34.548 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5700 IOs, 2199803904 bytes 2024-09-17T23:10:34.683 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5800 IOs, 2275363840 bytes 2024-09-17T23:10:34.805 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5900 IOs, 2344488960 bytes 2024-09-17T23:10:34.864 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6000 IOs, 2359463936 bytes 2024-09-17T23:10:34.975 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6100 IOs, 2407884800 bytes 2024-09-17T23:10:35.081 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6200 IOs, 2453066752 bytes 2024-09-17T23:10:35.158 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6300 IOs, 2468947968 bytes 2024-09-17T23:10:35.229 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6400 IOs, 2484836352 bytes 2024-09-17T23:10:35.299 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6500 IOs, 2492919808 bytes 2024-09-17T23:10:35.408 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6600 IOs, 2558243840 bytes 2024-09-17T23:10:35.494 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6700 IOs, 2581656576 bytes 2024-09-17T23:10:35.595 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6800 IOs, 2625581056 bytes 2024-09-17T23:10:35.677 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6900 IOs, 2651923456 bytes 2024-09-17T23:10:35.766 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7000 IOs, 2684894208 bytes 2024-09-17T23:10:35.833 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7100 IOs, 2692238336 bytes 2024-09-17T23:10:35.942 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7200 IOs, 2748395520 bytes 2024-09-17T23:10:36.028 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ error_code=124 2024-09-17T23:10:36.028 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set -e 2024-09-17T23:10:36.028 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' 124 -eq 124 ']' 2024-09-17T23:10:36.028 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:10:36.028 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-17T23:10:36.028 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:10:36.028 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:10:36.028 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:10:36.028 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state_pattern=up+replaying 2024-09-17T23:10:36.029 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local description_pattern= 2024-09-17T23:10:36.029 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local service_pattern= 2024-09-17T23:10:36.029 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:10:36.029 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:10:36.792 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-09-17T23:10:36.793 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-09-17T23:10:36.793 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2024-09-17T23:10:36.793 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-09-17T23:10:36.793 INFO:teuthology.orchestra.run:waiting for 600 2024-09-17T23:10:36.794 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi038.stderr:2024-09-17T23:10:36.791+0000 7fa11fc8b640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 36690) UID: 1000 2024-09-17T23:10:37.029 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-17T23:10:37.029 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:10:37.029 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:10:37.029 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:10:37.029 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state_pattern=up+replaying 2024-09-17T23:10:37.029 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local description_pattern= 2024-09-17T23:10:37.029 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local service_pattern= 2024-09-17T23:10:37.029 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status 2024-09-17T23:10:37.029 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ CEPH_ARGS= 2024-09-17T23:10:37.029 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-17T23:10:37.086 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ status='test: 2024-09-17T23:10:37.086 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: global_id: d6dfe272-2e04-4592-a96a-96648936ad18 2024-09-17T23:10:37.086 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: state: up+replaying 2024-09-17T23:10:37.086 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: description: replaying, {"bytes_per_second":6990506.67,"bytes_per_snapshot":535219200.0,"last_snapshot_bytes":535688192,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726614619,"remote_snapshot_timestamp":1726614619,"replay_state":"idle"} 2024-09-17T23:10:37.086 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: service: mirror.3 on smithi038 2024-09-17T23:10:37.086 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: last_update: 2024-09-17 23:10:25 2024-09-17T23:10:37.086 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: peer_sites: 2024-09-17T23:10:37.086 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: name: cluster2 2024-09-17T23:10:37.086 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: state: down+stopped 2024-09-17T23:10:37.086 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: description: local image is primary 2024-09-17T23:10:37.086 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: last_update: 2024-09-17 23:10:24' 2024-09-17T23:10:37.087 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep '^ state: .*up+replaying' 2024-09-17T23:10:37.087 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: state: up+replaying 2024-09-17T23:10:37.087 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep '^ description: .*' 2024-09-17T23:10:37.088 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: description: replaying, {"bytes_per_second":6990506.67,"bytes_per_snapshot":535219200.0,"last_snapshot_bytes":535688192,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726614619,"remote_snapshot_timestamp":1726614619,"replay_state":"idle"} 2024-09-17T23:10:37.089 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n '' ']' 2024-09-17T23:10:37.089 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ echo up+replaying 2024-09-17T23:10:37.089 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep '^up+' 2024-09-17T23:10:37.089 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:up+replaying 2024-09-17T23:10:37.090 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:10:37.091 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: service: mirror.3 on smithi038 2024-09-17T23:10:37.091 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local last_update 2024-09-17T23:10:37.091 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:10:37.093 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ last_update='2024-09-17 23:10:25' 2024-09-17T23:10:37.093 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-17 23:10:25' 2024-09-17T23:10:37.094 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:10:37.094 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:10:37.094 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:10:37.094 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state_pattern=up+replaying 2024-09-17T23:10:37.094 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local 'prev_last_update=2024-09-17 23:10:25' 2024-09-17T23:10:37.094 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status 2024-09-17T23:10:37.094 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ CEPH_ARGS= 2024-09-17T23:10:37.094 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-17T23:10:37.157 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ status='WARNINGOKWARNING142214298mirror.2smithi03819.1.1-330-gc2ddf972falseOK42234294mirror.3smithi03819.1.1-330-gc2ddf972trueOK42624310mirror.1smithi03819.1.1-330-gc2ddf972falseOK43374340mirror.0smithi03819.1.1-330-gc2ddf972falseOKtestd6dfe272-2e04-4592-a96a-96648936ad18up+replayingreplaying, {"bytes_per_second":6990506.67,"bytes_per_snapshot":535219200.0,"last_snapshot_bytes":535688192,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726614619,"remote_snapshot_timestamp":1726614619,"replay_state":"idle"}42234294mirror.3smithi0382024-09-17 23:10:25cluster20ba5654d-47fb-4672-b20c-7359a1340314down+stoppedlocal image is primary2024-09-17 23:10:24' 2024-09-17T23:10:37.158 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local last_update state 2024-09-17T23:10:37.158 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-17T23:10:37.159 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ last_update='2024-09-17 23:10:25' 2024-09-17T23:10:37.160 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-17T23:10:37.161 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ state=up+replaying 2024-09-17T23:10:37.162 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ echo up+replaying 2024-09-17T23:10:37.162 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep up+replaying 2024-09-17T23:10:37.162 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:up+replaying 2024-09-17T23:10:37.163 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:10:37.163 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:10:37.163 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ snap_name=snap5 2024-09-17T23:10:37.164 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_snap cluster2 mirror test snap5 2024-09-17T23:10:37.164 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:10:37.164 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:10:37.164 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:10:37.164 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap5 2024-09-17T23:10:37.164 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap5 --debug-rbd=20 --debug-journaler=20 2024-09-17T23:10:42.895 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-09-17T23:10:42.895 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-09-17T23:10:42.896 DEBUG:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-09-17T23:10:42.899 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-09-17T23:10:42.899 INFO:tasks.rbd_mirror_thrash:waiting for 94 secs before thrashing 2024-09-17T23:11:06.107 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-17T23:11:06.107 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:11:06.108 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:11:06.108 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:11:06.108 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-17T23:11:06.108 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:11:06.108 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:11:06.108 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:11:06.108 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=started 2024-09-17T23:11:06.108 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:11:06.108 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:11:06.108 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:11:07.108 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-17T23:11:07.108 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:11:07.108 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:11:07.108 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:11:07.108 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=started 2024-09-17T23:11:07.109 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_result 2024-09-17T23:11:07.109 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=stopped 2024-09-17T23:11:07.110 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-17T23:11:07.110 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_instance=cluster1 2024-09-17T23:11:07.110 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:11:07.110 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ grep -i state 2024-09-17T23:11:07.110 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:11:07.110 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance=cluster1 2024-09-17T23:11:07.110 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local loop_instance 2024-09-17T23:11:07.111 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:11:07.111 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sleep 0 2024-09-17T23:11:07.111 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-17T23:11:07.111 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ seq 0 3 2024-09-17T23:11:07.113 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:11:07.114 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-17T23:11:07.114 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:0 2024-09-17T23:11:07.114 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:11:07.114 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:11:07.114 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-17T23:11:07.114 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:0 2024-09-17T23:11:07.114 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:11:07.114 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:11:07.114 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:11:07.114 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=0 2024-09-17T23:11:07.114 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-17T23:11:07.114 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:11:07.114 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:11:07.115 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=0 2024-09-17T23:11:07.115 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:11:07.115 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-17T23:11:07.115 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:0 2024-09-17T23:11:07.115 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:11:07.115 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:11:07.115 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-17T23:11:07.115 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:0 2024-09-17T23:11:07.115 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:11:07.115 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:11:07.115 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:11:07.115 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:11:07.115 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-17T23:11:07.115 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:11:07.116 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:11:07.116 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=0 2024-09-17T23:11:07.116 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=0 2024-09-17T23:11:07.116 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-17T23:11:07.134 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:11:07.134 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:11:07.134 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:11:07.135 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-17T23:11:07.216 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-17T23:11:07.216 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd mirror status : get status for rbd mirror 2024-09-17T23:11:07.216 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:admin_socket: invalid command 2024-09-17T23:11:07.219 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:11:07.219 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-17T23:11:07.219 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:1 2024-09-17T23:11:07.219 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:11:07.219 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:11:07.219 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-17T23:11:07.220 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:1 2024-09-17T23:11:07.220 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:11:07.220 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:11:07.220 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:11:07.220 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=1 2024-09-17T23:11:07.220 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-17T23:11:07.220 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:11:07.220 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:11:07.220 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=1 2024-09-17T23:11:07.220 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=1 2024-09-17T23:11:07.220 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-17T23:11:07.220 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:1 2024-09-17T23:11:07.220 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:11:07.221 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:11:07.221 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-17T23:11:07.221 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:1 2024-09-17T23:11:07.221 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:11:07.221 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:11:07.221 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:11:07.221 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=1 2024-09-17T23:11:07.221 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-17T23:11:07.221 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:11:07.221 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:11:07.221 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=1 2024-09-17T23:11:07.221 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=1 2024-09-17T23:11:07.221 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-17T23:11:07.238 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-17T23:11:07.238 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-17T23:11:07.239 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-17T23:11:07.239 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-17T23:11:07.315 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-17T23:11:07.315 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd mirror status : get status for rbd mirror 2024-09-17T23:11:07.315 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:admin_socket: invalid command 2024-09-17T23:11:07.318 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:11:07.318 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-17T23:11:07.318 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:2 2024-09-17T23:11:07.318 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:11:07.318 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:11:07.318 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-17T23:11:07.318 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:2 2024-09-17T23:11:07.318 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:11:07.318 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:11:07.318 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:11:07.318 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=2 2024-09-17T23:11:07.318 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-17T23:11:07.318 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:11:07.318 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:11:07.319 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=2 2024-09-17T23:11:07.319 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=2 2024-09-17T23:11:07.319 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-17T23:11:07.319 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:2 2024-09-17T23:11:07.319 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:11:07.319 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:11:07.319 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-17T23:11:07.319 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:2 2024-09-17T23:11:07.319 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:11:07.319 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:11:07.319 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:11:07.319 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=2 2024-09-17T23:11:07.319 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-17T23:11:07.319 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:11:07.319 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:11:07.319 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=2 2024-09-17T23:11:07.319 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=2 2024-09-17T23:11:07.320 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-17T23:11:07.336 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-17T23:11:07.336 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-17T23:11:07.336 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-17T23:11:07.336 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-17T23:11:07.412 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-17T23:11:07.412 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd mirror status : get status for rbd mirror 2024-09-17T23:11:07.412 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:admin_socket: invalid command 2024-09-17T23:11:07.414 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:11:07.414 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-09-17T23:11:07.414 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:3 2024-09-17T23:11:07.415 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:11:07.415 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:11:07.415 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-17T23:11:07.415 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:3 2024-09-17T23:11:07.415 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:11:07.415 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:11:07.415 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:11:07.415 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=3 2024-09-17T23:11:07.415 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-17T23:11:07.415 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:11:07.415 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:11:07.415 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=3 2024-09-17T23:11:07.415 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=3 2024-09-17T23:11:07.415 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-17T23:11:07.416 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:3 2024-09-17T23:11:07.416 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:11:07.416 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:11:07.416 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-17T23:11:07.416 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:3 2024-09-17T23:11:07.416 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:11:07.416 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:11:07.416 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:11:07.416 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=3 2024-09-17T23:11:07.416 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-17T23:11:07.416 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:11:07.416 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:11:07.416 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=3 2024-09-17T23:11:07.416 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=3 2024-09-17T23:11:07.416 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-17T23:11:07.434 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-17T23:11:07.434 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-17T23:11:07.434 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-17T23:11:07.434 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-09-17T23:11:07.518 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ return 0 2024-09-17T23:11:07.518 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ status_result=' "state": "Replaying"' 2024-09-17T23:11:07.518 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ echo ' "state": "Replaying"' 2024-09-17T23:11:07.519 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep -i Replaying 2024-09-17T23:11:07.519 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: "state": "Replaying" 2024-09-17T23:11:07.521 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_state=started 2024-09-17T23:11:07.521 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test started = started 2024-09-17T23:11:07.521 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:11:07.521 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-17T23:11:07.521 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:11:07.521 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:11:07.521 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:11:07.521 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:11:07.521 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = journal ']' 2024-09-17T23:11:07.521 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = snapshot ']' 2024-09-17T23:11:07.521 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-09-17T23:11:07.522 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:11:07.522 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:11:07.522 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:11:07.522 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:11:07.522 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster2-mirror-test.status 2024-09-17T23:11:07.522 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster2-mirror-test.status 2024-09-17T23:11:07.522 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:11:07.523 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-test.status 2024-09-17T23:11:07.523 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster1-mirror-test.status 2024-09-17T23:11:07.524 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster1-mirror-test.status 2024-09-17T23:11:07.524 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:11:07.524 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_status_log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-test.status 2024-09-17T23:11:07.524 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-17T23:11:07.524 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:11:07.524 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:11:07.524 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:11:07.524 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-17T23:11:09.119 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Snapshot ID: 17 2024-09-17T23:11:09.145 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.status 2024-09-17T23:11:09.145 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:11:09.145 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:11:09.145 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:11:09.145 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-test.status 2024-09-17T23:11:09.145 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-17T23:11:09.145 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:11:09.216 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-17T23:11:09.218 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snapshot_id=17 2024-09-17T23:11:09.218 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:11:09.218 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:11:09.218 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.2 2024-09-17T23:11:09.420 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.status 2024-09-17T23:11:09.420 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:11:09.420 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:11:09.420 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:11:09.420 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-test.status 2024-09-17T23:11:09.420 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-17T23:11:09.420 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:11:09.473 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:11:09.474 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:11:09.475 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:11:09.475 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:11:09.475 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:11:09.476 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:11:09.476 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 17 = '' 2024-09-17T23:11:09.476 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:11:09.476 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.4 2024-09-17T23:11:09.878 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.status 2024-09-17T23:11:09.878 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:11:09.878 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:11:09.878 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:11:09.878 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-test.status 2024-09-17T23:11:09.878 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-17T23:11:09.878 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:11:09.932 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:11:09.932 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:11:09.933 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:11:09.934 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:11:09.934 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:11:09.934 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:11:09.934 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 17 = '' 2024-09-17T23:11:09.934 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:11:09.935 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.8 2024-09-17T23:11:10.736 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.status 2024-09-17T23:11:10.736 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:11:10.736 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:11:10.736 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:11:10.736 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-test.status 2024-09-17T23:11:10.736 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-17T23:11:10.736 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:11:10.790 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:11:10.790 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:11:10.792 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:11:10.792 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:11:10.792 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:11:10.793 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:11:10.793 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 17 = '' 2024-09-17T23:11:10.793 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:11:10.793 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1.6 2024-09-17T23:11:12.394 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.status 2024-09-17T23:11:12.394 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:11:12.394 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:11:12.395 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:11:12.395 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-test.status 2024-09-17T23:11:12.395 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-17T23:11:12.395 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:11:12.445 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:11:12.448 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id=17 2024-09-17T23:11:12.448 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 17 = 17 2024-09-17T23:11:12.448 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:11:12.448 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snap_present cluster1 mirror test snap5 2024-09-17T23:11:12.448 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:11:12.448 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:11:12.448 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:11:12.448 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap5 2024-09-17T23:11:12.448 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:11:12.448 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-17T23:11:12.448 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:11:13.449 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 info mirror/test@snap5 2024-09-17T23:11:13.498 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'test': 2024-09-17T23:11:13.498 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 512 MiB in 128 objects 2024-09-17T23:11:13.498 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:11:13.498 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 10 2024-09-17T23:11:13.499 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 10c7e03f2edd 2024-09-17T23:11:13.499 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.10c7e03f2edd 2024-09-17T23:11:13.499 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:11:13.499 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-17T23:11:13.499 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:11:13.499 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:11:13.499 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:06:57 2024 2024-09-17T23:11:13.499 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:10:25 2024 2024-09-17T23:11:13.499 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:06:57 2024 2024-09-17T23:11:13.499 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: protected: False 2024-09-17T23:11:13.499 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:11:13.499 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:11:13.499 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: d6dfe272-2e04-4592-a96a-96648936ad18 2024-09-17T23:11:13.499 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: false 2024-09-17T23:11:13.502 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:11:13.502 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n snap4-clean ']' 2024-09-17T23:11:13.502 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ compare_image_snaps mirror test snap4-clean 2024-09-17T23:11:13.502 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:11:13.502 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:11:13.502 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap4-clean 2024-09-17T23:11:13.503 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local ret=0 2024-09-17T23:11:13.503 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local rmt_export=/tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export 2024-09-17T23:11:13.503 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local loc_export=/tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:11:13.503 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:11:13.503 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4-clean /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export 2024-09-17T23:11:14.023 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-17T23:11:14.034 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4-clean /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:11:14.686 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-17T23:11:14.694 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ cmp /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:11:14.939 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:11:15.060 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:11:15.061 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ compare_image_snaps mirror test snap5 2024-09-17T23:11:15.061 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:11:15.061 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:11:15.061 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap5 2024-09-17T23:11:15.061 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local ret=0 2024-09-17T23:11:15.061 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local rmt_export=/tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export 2024-09-17T23:11:15.061 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local loc_export=/tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:11:15.061 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:11:15.061 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5 /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export 2024-09-17T23:11:15.573 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-17T23:11:15.584 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5 /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:11:16.269 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-17T23:11:16.276 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ cmp /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:11:16.523 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:11:16.642 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:11:16.642 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ clean_snap_name=snap5-clean 2024-09-17T23:11:16.642 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_snap cluster2 mirror test snap5-clean 2024-09-17T23:11:16.642 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:11:16.642 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:11:16.642 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:11:16.642 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap5-clean 2024-09-17T23:11:16.643 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap5-clean --debug-rbd=20 --debug-journaler=20 2024-09-17T23:11:17.029 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 10` 2024-09-17T23:11:17.029 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ stress_write_image cluster2 mirror test 2024-09-17T23:11:17.029 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:11:17.029 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:11:17.029 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:11:17.029 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-17T23:11:17.031 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local duration=10 2024-09-17T23:11:17.031 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set +e 2024-09-17T23:11:17.031 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-17T23:11:17.315 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:100 IOs, 46241792 bytes 2024-09-17T23:11:17.458 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:200 IOs, 84547584 bytes 2024-09-17T23:11:17.582 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:300 IOs, 132975616 bytes 2024-09-17T23:11:17.680 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:400 IOs, 157924352 bytes 2024-09-17T23:11:17.787 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:500 IOs, 198617088 bytes 2024-09-17T23:11:17.939 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:600 IOs, 297729024 bytes 2024-09-17T23:11:18.023 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:700 IOs, 319053824 bytes 2024-09-17T23:11:18.106 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:800 IOs, 350275584 bytes 2024-09-17T23:11:18.211 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:900 IOs, 398338048 bytes 2024-09-17T23:11:18.321 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1000 IOs, 450579456 bytes 2024-09-17T23:11:18.410 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1100 IOs, 490937344 bytes 2024-09-17T23:11:18.496 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1200 IOs, 510910464 bytes 2024-09-17T23:11:18.651 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1300 IOs, 621554688 bytes 2024-09-17T23:11:18.714 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1400 IOs, 639203328 bytes 2024-09-17T23:11:18.799 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1500 IOs, 653515776 bytes 2024-09-17T23:11:18.888 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1600 IOs, 684649472 bytes 2024-09-17T23:11:18.978 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1700 IOs, 719255552 bytes 2024-09-17T23:11:19.094 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1800 IOs, 778263552 bytes 2024-09-17T23:11:19.187 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1900 IOs, 812652544 bytes 2024-09-17T23:11:19.268 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2000 IOs, 835593216 bytes 2024-09-17T23:11:19.386 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2100 IOs, 901174272 bytes 2024-09-17T23:11:19.495 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2200 IOs, 951145472 bytes 2024-09-17T23:11:19.604 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2300 IOs, 1001777152 bytes 2024-09-17T23:11:19.678 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2400 IOs, 1018632192 bytes 2024-09-17T23:11:19.805 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2500 IOs, 1101478912 bytes 2024-09-17T23:11:19.960 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2600 IOs, 1197287424 bytes 2024-09-17T23:11:20.040 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2700 IOs, 1223883776 bytes 2024-09-17T23:11:20.143 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2800 IOs, 1263345664 bytes 2024-09-17T23:11:20.275 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2900 IOs, 1349432320 bytes 2024-09-17T23:11:20.443 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3000 IOs, 1458165760 bytes 2024-09-17T23:11:20.561 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3100 IOs, 1520951296 bytes 2024-09-17T23:11:20.693 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3200 IOs, 1600683008 bytes 2024-09-17T23:11:20.774 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3300 IOs, 1621862400 bytes 2024-09-17T23:11:20.843 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3400 IOs, 1629919232 bytes 2024-09-17T23:11:20.971 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3500 IOs, 1707542528 bytes 2024-09-17T23:11:21.041 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3600 IOs, 1708171264 bytes 2024-09-17T23:11:21.108 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3700 IOs, 1715258368 bytes 2024-09-17T23:11:21.214 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3800 IOs, 1762232320 bytes 2024-09-17T23:11:21.319 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3900 IOs, 1825959936 bytes 2024-09-17T23:11:21.414 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4000 IOs, 1857243136 bytes 2024-09-17T23:11:21.499 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4100 IOs, 1878630400 bytes 2024-09-17T23:11:21.594 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4200 IOs, 1914926080 bytes 2024-09-17T23:11:21.671 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4300 IOs, 1934015488 bytes 2024-09-17T23:11:21.755 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4400 IOs, 1964579840 bytes 2024-09-17T23:11:21.789 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-09-17T23:11:21.789 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-09-17T23:11:21.790 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2024-09-17T23:11:21.790 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi038.stderr:2024-09-17T23:11:21.788+0000 7f52e6412640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 38899) UID: 1000 2024-09-17T23:11:21.877 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4500 IOs, 2025451520 bytes 2024-09-17T23:11:21.970 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4600 IOs, 2060676096 bytes 2024-09-17T23:11:22.062 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4700 IOs, 2097671168 bytes 2024-09-17T23:11:22.147 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4800 IOs, 2119035904 bytes 2024-09-17T23:11:22.271 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4900 IOs, 2192876544 bytes 2024-09-17T23:11:22.385 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5000 IOs, 2246281216 bytes 2024-09-17T23:11:22.448 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5100 IOs, 2252924928 bytes 2024-09-17T23:11:22.530 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5200 IOs, 2276097024 bytes 2024-09-17T23:11:22.648 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5300 IOs, 2345110528 bytes 2024-09-17T23:11:22.731 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5400 IOs, 2366790656 bytes 2024-09-17T23:11:22.809 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5500 IOs, 2386977792 bytes 2024-09-17T23:11:22.871 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5600 IOs, 2387598336 bytes 2024-09-17T23:11:22.958 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5700 IOs, 2424007680 bytes 2024-09-17T23:11:23.055 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5800 IOs, 2449384448 bytes 2024-09-17T23:11:23.142 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5900 IOs, 2476537856 bytes 2024-09-17T23:11:23.278 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6000 IOs, 2557296640 bytes 2024-09-17T23:11:23.351 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6100 IOs, 2577033216 bytes 2024-09-17T23:11:23.472 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6200 IOs, 2631075840 bytes 2024-09-17T23:11:23.597 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6300 IOs, 2701284352 bytes 2024-09-17T23:11:23.682 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6400 IOs, 2727072768 bytes 2024-09-17T23:11:23.762 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6500 IOs, 2753601536 bytes 2024-09-17T23:11:23.842 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6600 IOs, 2770967552 bytes 2024-09-17T23:11:23.913 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6700 IOs, 2788975616 bytes 2024-09-17T23:11:24.004 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6800 IOs, 2813630464 bytes 2024-09-17T23:11:24.122 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6900 IOs, 2874462208 bytes 2024-09-17T23:11:24.233 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7000 IOs, 2928788480 bytes 2024-09-17T23:11:24.317 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7100 IOs, 2950232064 bytes 2024-09-17T23:11:24.395 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7200 IOs, 2968678400 bytes 2024-09-17T23:11:24.503 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7300 IOs, 3015521280 bytes 2024-09-17T23:11:24.631 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7400 IOs, 3090909184 bytes 2024-09-17T23:11:24.738 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7500 IOs, 3131009024 bytes 2024-09-17T23:11:24.875 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7600 IOs, 3217430528 bytes 2024-09-17T23:11:24.988 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7700 IOs, 3268448256 bytes 2024-09-17T23:11:25.091 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7800 IOs, 3307767808 bytes 2024-09-17T23:11:25.166 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7900 IOs, 3320047616 bytes 2024-09-17T23:11:25.265 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8000 IOs, 3357531136 bytes 2024-09-17T23:11:25.330 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8100 IOs, 3359646720 bytes 2024-09-17T23:11:25.440 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8200 IOs, 3410007040 bytes 2024-09-17T23:11:25.562 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8300 IOs, 3473810432 bytes 2024-09-17T23:11:25.649 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8400 IOs, 3496853504 bytes 2024-09-17T23:11:25.743 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8500 IOs, 3533620224 bytes 2024-09-17T23:11:25.790 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-09-17T23:11:25.790 INFO:teuthology.orchestra.run:waiting for 600 2024-09-17T23:11:25.790 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-09-17T23:11:25.791 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-09-17T23:11:25.791 DEBUG:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-09-17T23:11:25.797 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-09-17T23:11:25.797 INFO:tasks.rbd_mirror_thrash:waiting for 104 secs before thrashing 2024-09-17T23:11:25.854 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8600 IOs, 3578961920 bytes 2024-09-17T23:11:25.961 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8700 IOs, 3612902400 bytes 2024-09-17T23:11:26.037 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8800 IOs, 3624767488 bytes 2024-09-17T23:11:26.164 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8900 IOs, 3671934976 bytes 2024-09-17T23:11:26.290 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9000 IOs, 3750136832 bytes 2024-09-17T23:11:26.411 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9100 IOs, 3803685888 bytes 2024-09-17T23:11:26.476 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9200 IOs, 3806647296 bytes 2024-09-17T23:11:26.564 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9300 IOs, 3844804608 bytes 2024-09-17T23:11:26.669 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9400 IOs, 3883592704 bytes 2024-09-17T23:11:26.743 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9500 IOs, 3915478016 bytes 2024-09-17T23:11:26.849 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9600 IOs, 3950630912 bytes 2024-09-17T23:11:26.982 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9700 IOs, 4012280832 bytes 2024-09-17T23:11:27.036 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ error_code=124 2024-09-17T23:11:27.037 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set -e 2024-09-17T23:11:27.037 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' 124 -eq 124 ']' 2024-09-17T23:11:27.037 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:11:27.037 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-17T23:11:27.037 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:11:27.037 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:11:27.037 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:11:27.037 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state_pattern=up+replaying 2024-09-17T23:11:27.037 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local description_pattern= 2024-09-17T23:11:27.037 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local service_pattern= 2024-09-17T23:11:27.037 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:11:27.037 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:11:28.038 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-17T23:11:28.038 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:11:28.038 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:11:28.038 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:11:28.038 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state_pattern=up+replaying 2024-09-17T23:11:28.038 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local description_pattern= 2024-09-17T23:11:28.038 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local service_pattern= 2024-09-17T23:11:28.038 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status 2024-09-17T23:11:28.038 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ CEPH_ARGS= 2024-09-17T23:11:28.038 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-17T23:11:28.100 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ status='test: 2024-09-17T23:11:28.101 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: global_id: d6dfe272-2e04-4592-a96a-96648936ad18 2024-09-17T23:11:28.101 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: state: up+replaying 2024-09-17T23:11:28.101 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: description: replaying, {"bytes_per_second":17504358.4,"bytes_per_snapshot":530409472.0,"last_snapshot_bytes":525130752,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726614668,"remote_snapshot_timestamp":1726614668,"replay_state":"idle"} 2024-09-17T23:11:28.101 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: service: mirror.3 on smithi038 2024-09-17T23:11:28.101 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: last_update: 2024-09-17 23:11:25 2024-09-17T23:11:28.101 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: peer_sites: 2024-09-17T23:11:28.101 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: name: cluster2 2024-09-17T23:11:28.101 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: state: up+stopped 2024-09-17T23:11:28.101 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: description: local image is primary 2024-09-17T23:11:28.101 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: last_update: 2024-09-17 23:11:24' 2024-09-17T23:11:28.101 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep '^ state: .*up+replaying' 2024-09-17T23:11:28.102 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: state: up+replaying 2024-09-17T23:11:28.102 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep '^ description: .*' 2024-09-17T23:11:28.103 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: description: replaying, {"bytes_per_second":17504358.4,"bytes_per_snapshot":530409472.0,"last_snapshot_bytes":525130752,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726614668,"remote_snapshot_timestamp":1726614668,"replay_state":"idle"} 2024-09-17T23:11:28.103 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n '' ']' 2024-09-17T23:11:28.103 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ echo up+replaying 2024-09-17T23:11:28.103 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep '^up+' 2024-09-17T23:11:28.104 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:up+replaying 2024-09-17T23:11:28.104 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:11:28.105 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: service: mirror.3 on smithi038 2024-09-17T23:11:28.106 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local last_update 2024-09-17T23:11:28.106 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:11:28.108 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ last_update='2024-09-17 23:11:25' 2024-09-17T23:11:28.108 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-17 23:11:25' 2024-09-17T23:11:28.108 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:11:28.108 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:11:28.108 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:11:28.108 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state_pattern=up+replaying 2024-09-17T23:11:28.109 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local 'prev_last_update=2024-09-17 23:11:25' 2024-09-17T23:11:28.109 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status 2024-09-17T23:11:28.109 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ CEPH_ARGS= 2024-09-17T23:11:28.109 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-17T23:11:28.169 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ status='WARNINGWARNINGOK142214298mirror.2smithi03819.1.1-330-gc2ddf972falseOK42234294mirror.3smithi03819.1.1-330-gc2ddf972trueOK42624310mirror.1smithi03819.1.1-330-gc2ddf972falseOK4483mirror.0smithi03819.1.1-330-gc2ddf972falseWARNINGnot reporting statustestd6dfe272-2e04-4592-a96a-96648936ad18up+replayingreplaying, {"bytes_per_second":17504358.4,"bytes_per_snapshot":530409472.0,"last_snapshot_bytes":525130752,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726614668,"remote_snapshot_timestamp":1726614668,"replay_state":"idle"}42234294mirror.3smithi0382024-09-17 23:11:25cluster20ba5654d-47fb-4672-b20c-7359a1340314up+stoppedlocal image is primary2024-09-17 23:11:24' 2024-09-17T23:11:28.170 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local last_update state 2024-09-17T23:11:28.170 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-17T23:11:28.172 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ last_update='2024-09-17 23:11:25' 2024-09-17T23:11:28.172 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-17T23:11:28.174 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ state=up+replaying 2024-09-17T23:11:28.174 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ echo up+replaying 2024-09-17T23:11:28.174 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep up+replaying 2024-09-17T23:11:28.175 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:up+replaying 2024-09-17T23:11:28.176 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:11:28.176 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:11:28.176 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ snap_name=snap6 2024-09-17T23:11:28.176 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_snap cluster2 mirror test snap6 2024-09-17T23:11:28.176 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:11:28.176 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:11:28.176 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:11:28.176 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap6 2024-09-17T23:11:28.176 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap6 --debug-rbd=20 --debug-journaler=20 2024-09-17T23:11:58.090 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-17T23:11:58.090 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:11:58.090 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:11:58.090 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:11:58.090 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-17T23:11:58.090 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:11:58.090 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:11:58.090 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:11:58.090 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=started 2024-09-17T23:11:58.090 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:11:58.091 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:11:58.091 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:11:59.096 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-17T23:11:59.096 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:11:59.096 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:11:59.096 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:11:59.096 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=started 2024-09-17T23:11:59.096 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_result 2024-09-17T23:11:59.096 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=stopped 2024-09-17T23:11:59.096 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-17T23:11:59.097 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_instance=cluster1 2024-09-17T23:11:59.097 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ grep -i state 2024-09-17T23:11:59.097 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:11:59.097 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:11:59.097 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance=cluster1 2024-09-17T23:11:59.097 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local loop_instance 2024-09-17T23:11:59.097 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:11:59.097 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sleep 0 2024-09-17T23:11:59.097 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-17T23:11:59.097 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ seq 0 3 2024-09-17T23:11:59.097 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:11:59.097 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-17T23:11:59.097 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:0 2024-09-17T23:11:59.097 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:11:59.098 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:11:59.098 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-17T23:11:59.098 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:0 2024-09-17T23:11:59.098 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:11:59.098 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:11:59.098 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:11:59.098 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=0 2024-09-17T23:11:59.098 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-17T23:11:59.098 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:11:59.098 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:11:59.098 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=0 2024-09-17T23:11:59.098 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:11:59.098 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-17T23:11:59.098 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:0 2024-09-17T23:11:59.099 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:11:59.099 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:11:59.099 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-17T23:11:59.099 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:0 2024-09-17T23:11:59.099 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:11:59.099 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:11:59.099 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:11:59.099 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:11:59.099 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-17T23:11:59.099 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:11:59.099 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:11:59.099 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=0 2024-09-17T23:11:59.099 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=0 2024-09-17T23:11:59.099 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-17T23:11:59.118 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:11:59.118 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:11:59.118 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:11:59.118 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-17T23:11:59.200 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-17T23:11:59.200 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd mirror status : get status for rbd mirror 2024-09-17T23:11:59.200 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:admin_socket: invalid command 2024-09-17T23:11:59.203 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:11:59.203 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-17T23:11:59.203 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:1 2024-09-17T23:11:59.204 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:11:59.204 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:11:59.204 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-17T23:11:59.204 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:1 2024-09-17T23:11:59.204 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:11:59.204 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:11:59.204 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:11:59.204 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=1 2024-09-17T23:11:59.204 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-17T23:11:59.204 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:11:59.204 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:11:59.204 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=1 2024-09-17T23:11:59.204 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=1 2024-09-17T23:11:59.204 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-17T23:11:59.205 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:1 2024-09-17T23:11:59.205 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:11:59.205 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:11:59.205 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-17T23:11:59.205 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:1 2024-09-17T23:11:59.205 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:11:59.205 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:11:59.205 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:11:59.205 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=1 2024-09-17T23:11:59.205 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-17T23:11:59.205 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:11:59.206 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:11:59.206 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=1 2024-09-17T23:11:59.206 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=1 2024-09-17T23:11:59.206 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-17T23:11:59.223 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-17T23:11:59.223 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-17T23:11:59.223 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-17T23:11:59.223 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-17T23:11:59.299 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-17T23:11:59.299 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd mirror status : get status for rbd mirror 2024-09-17T23:11:59.299 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:admin_socket: invalid command 2024-09-17T23:11:59.302 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:11:59.302 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-17T23:11:59.302 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:2 2024-09-17T23:11:59.302 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:11:59.302 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:11:59.302 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-17T23:11:59.303 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:2 2024-09-17T23:11:59.303 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:11:59.303 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:11:59.303 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:11:59.303 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=2 2024-09-17T23:11:59.303 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-17T23:11:59.303 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:11:59.303 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:11:59.303 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=2 2024-09-17T23:11:59.303 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=2 2024-09-17T23:11:59.303 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-17T23:11:59.303 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:2 2024-09-17T23:11:59.304 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:11:59.304 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:11:59.304 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-17T23:11:59.304 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:2 2024-09-17T23:11:59.304 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:11:59.304 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:11:59.304 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:11:59.304 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=2 2024-09-17T23:11:59.304 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-17T23:11:59.304 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:11:59.305 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:11:59.305 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=2 2024-09-17T23:11:59.305 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=2 2024-09-17T23:11:59.305 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-17T23:11:59.321 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-17T23:11:59.321 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-17T23:11:59.321 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-17T23:11:59.321 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-17T23:11:59.396 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-17T23:11:59.396 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd mirror status : get status for rbd mirror 2024-09-17T23:11:59.396 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:admin_socket: invalid command 2024-09-17T23:11:59.398 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:11:59.398 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-09-17T23:11:59.399 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:3 2024-09-17T23:11:59.399 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:11:59.399 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:11:59.399 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-17T23:11:59.399 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:3 2024-09-17T23:11:59.399 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:11:59.399 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:11:59.399 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:11:59.399 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=3 2024-09-17T23:11:59.399 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-17T23:11:59.399 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:11:59.399 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:11:59.399 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=3 2024-09-17T23:11:59.399 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=3 2024-09-17T23:11:59.400 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-17T23:11:59.400 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:3 2024-09-17T23:11:59.400 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:11:59.400 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:11:59.400 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-17T23:11:59.400 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:3 2024-09-17T23:11:59.400 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:11:59.400 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:11:59.400 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:11:59.400 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=3 2024-09-17T23:11:59.400 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-17T23:11:59.400 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:11:59.400 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:11:59.400 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=3 2024-09-17T23:11:59.401 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=3 2024-09-17T23:11:59.402 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-17T23:11:59.417 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-17T23:11:59.418 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-17T23:11:59.418 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-17T23:11:59.418 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-09-17T23:11:59.501 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ return 0 2024-09-17T23:11:59.502 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ status_result=' "state": "Replaying"' 2024-09-17T23:11:59.502 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ echo ' "state": "Replaying"' 2024-09-17T23:11:59.502 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep -i Replaying 2024-09-17T23:11:59.503 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: "state": "Replaying" 2024-09-17T23:11:59.504 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_state=started 2024-09-17T23:11:59.504 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test started = started 2024-09-17T23:11:59.505 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:11:59.505 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-17T23:11:59.505 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:11:59.505 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:11:59.505 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:11:59.505 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:11:59.505 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = journal ']' 2024-09-17T23:11:59.505 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = snapshot ']' 2024-09-17T23:11:59.505 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-09-17T23:11:59.505 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:11:59.505 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:11:59.505 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:11:59.505 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:11:59.506 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster2-mirror-test.status 2024-09-17T23:11:59.506 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster2-mirror-test.status 2024-09-17T23:11:59.506 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:11:59.506 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-test.status 2024-09-17T23:11:59.506 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster1-mirror-test.status 2024-09-17T23:11:59.506 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster1-mirror-test.status 2024-09-17T23:11:59.507 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:11:59.507 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_status_log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-test.status 2024-09-17T23:11:59.507 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-17T23:11:59.507 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:11:59.507 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:11:59.508 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:11:59.508 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-17T23:12:01.097 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Snapshot ID: 20 2024-09-17T23:12:01.126 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.status 2024-09-17T23:12:01.126 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:12:01.126 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:12:01.126 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:12:01.126 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-test.status 2024-09-17T23:12:01.126 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-17T23:12:01.126 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:12:01.201 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-17T23:12:01.203 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snapshot_id=20 2024-09-17T23:12:01.203 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:12:01.203 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:12:01.203 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.2 2024-09-17T23:12:01.405 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.status 2024-09-17T23:12:01.405 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:12:01.405 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:12:01.405 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:12:01.405 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-test.status 2024-09-17T23:12:01.405 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-17T23:12:01.405 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:12:01.458 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:12:01.458 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:12:01.459 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:12:01.459 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:12:01.459 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:12:01.460 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:12:01.460 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 20 = '' 2024-09-17T23:12:01.460 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:12:01.460 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.4 2024-09-17T23:12:01.861 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.status 2024-09-17T23:12:01.861 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:12:01.862 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:12:01.862 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:12:01.862 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-test.status 2024-09-17T23:12:01.862 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-17T23:12:01.862 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:12:01.916 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:12:01.917 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:12:01.918 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:12:01.918 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:12:01.918 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:12:01.919 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:12:01.919 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 20 = '' 2024-09-17T23:12:01.919 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:12:01.919 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.8 2024-09-17T23:12:02.721 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.status 2024-09-17T23:12:02.721 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:12:02.721 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:12:02.721 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:12:02.721 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-test.status 2024-09-17T23:12:02.721 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-17T23:12:02.721 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:12:02.801 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:12:02.802 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:12:02.803 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:12:02.804 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:12:02.804 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:12:02.804 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:12:02.804 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 20 = '' 2024-09-17T23:12:02.804 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:12:02.805 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1.6 2024-09-17T23:12:04.406 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.status 2024-09-17T23:12:04.406 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:12:04.406 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:12:04.407 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:12:04.407 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-test.status 2024-09-17T23:12:04.407 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-17T23:12:04.407 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:12:04.464 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:12:04.467 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id=20 2024-09-17T23:12:04.467 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 20 = 20 2024-09-17T23:12:04.467 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:12:04.467 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snap_present cluster1 mirror test snap6 2024-09-17T23:12:04.467 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:12:04.467 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:12:04.467 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:12:04.467 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap6 2024-09-17T23:12:04.467 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:12:04.468 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-17T23:12:04.468 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:12:05.468 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 info mirror/test@snap6 2024-09-17T23:12:05.519 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'test': 2024-09-17T23:12:05.519 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 512 MiB in 128 objects 2024-09-17T23:12:05.519 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:12:05.519 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 12 2024-09-17T23:12:05.519 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 10c7e03f2edd 2024-09-17T23:12:05.519 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.10c7e03f2edd 2024-09-17T23:12:05.519 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:12:05.519 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-17T23:12:05.519 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:12:05.520 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:12:05.520 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:06:57 2024 2024-09-17T23:12:05.520 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:10:25 2024 2024-09-17T23:12:05.520 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:06:57 2024 2024-09-17T23:12:05.520 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: protected: False 2024-09-17T23:12:05.520 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:12:05.520 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:12:05.520 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: d6dfe272-2e04-4592-a96a-96648936ad18 2024-09-17T23:12:05.520 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: false 2024-09-17T23:12:05.523 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:12:05.523 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n snap5-clean ']' 2024-09-17T23:12:05.523 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ compare_image_snaps mirror test snap5-clean 2024-09-17T23:12:05.523 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:12:05.524 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:12:05.524 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap5-clean 2024-09-17T23:12:05.524 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local ret=0 2024-09-17T23:12:05.524 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local rmt_export=/tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export 2024-09-17T23:12:05.524 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local loc_export=/tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:12:05.524 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:12:05.524 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5-clean /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export 2024-09-17T23:12:06.024 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-17T23:12:06.033 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5-clean /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:12:06.726 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-17T23:12:06.733 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ cmp /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:12:06.979 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:12:07.103 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:12:07.103 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ compare_image_snaps mirror test snap6 2024-09-17T23:12:07.103 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:12:07.103 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:12:07.103 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap6 2024-09-17T23:12:07.103 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local ret=0 2024-09-17T23:12:07.103 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local rmt_export=/tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export 2024-09-17T23:12:07.103 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local loc_export=/tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:12:07.103 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:12:07.104 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6 /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export 2024-09-17T23:12:07.617 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-17T23:12:07.628 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6 /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:12:08.308 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-17T23:12:08.316 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ cmp /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:12:08.561 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:12:08.681 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:12:08.681 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ clean_snap_name=snap6-clean 2024-09-17T23:12:08.681 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_snap cluster2 mirror test snap6-clean 2024-09-17T23:12:08.681 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:12:08.682 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:12:08.682 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:12:08.682 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap6-clean 2024-09-17T23:12:08.682 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap6-clean --debug-rbd=20 --debug-journaler=20 2024-09-17T23:12:09.035 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 10` 2024-09-17T23:12:09.035 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ stress_write_image cluster2 mirror test 2024-09-17T23:12:09.035 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:12:09.035 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:12:09.035 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:12:09.035 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-17T23:12:09.037 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local duration=7 2024-09-17T23:12:09.037 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set +e 2024-09-17T23:12:09.037 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-17T23:12:09.338 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:100 IOs, 57156608 bytes 2024-09-17T23:12:09.501 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:200 IOs, 103200768 bytes 2024-09-17T23:12:09.622 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:300 IOs, 124471296 bytes 2024-09-17T23:12:09.703 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:400 IOs, 137764864 bytes 2024-09-17T23:12:09.785 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:500 IOs, 153985024 bytes 2024-09-17T23:12:09.899 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:600 IOs, 205701120 bytes 2024-09-17T23:12:10.001 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:700 IOs, 247101440 bytes 2024-09-17T23:12:10.071 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:800 IOs, 254569472 bytes 2024-09-17T23:12:10.149 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:900 IOs, 269663232 bytes 2024-09-17T23:12:10.212 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1000 IOs, 270286848 bytes 2024-09-17T23:12:10.329 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1100 IOs, 322578432 bytes 2024-09-17T23:12:10.465 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1200 IOs, 400834560 bytes 2024-09-17T23:12:10.580 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1300 IOs, 458925056 bytes 2024-09-17T23:12:10.721 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1400 IOs, 547236864 bytes 2024-09-17T23:12:10.806 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1500 IOs, 568404992 bytes 2024-09-17T23:12:10.872 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1600 IOs, 568984576 bytes 2024-09-17T23:12:10.987 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1700 IOs, 626802688 bytes 2024-09-17T23:12:11.090 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1800 IOs, 677498880 bytes 2024-09-17T23:12:11.191 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1900 IOs, 702038016 bytes 2024-09-17T23:12:11.296 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2000 IOs, 744287232 bytes 2024-09-17T23:12:11.414 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2100 IOs, 807532544 bytes 2024-09-17T23:12:11.536 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2200 IOs, 872153088 bytes 2024-09-17T23:12:11.667 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2300 IOs, 949710848 bytes 2024-09-17T23:12:11.782 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2400 IOs, 994844672 bytes 2024-09-17T23:12:11.864 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2500 IOs, 1013711872 bytes 2024-09-17T23:12:11.979 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2600 IOs, 1066020864 bytes 2024-09-17T23:12:12.089 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2700 IOs, 1113152512 bytes 2024-09-17T23:12:12.199 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2800 IOs, 1165095936 bytes 2024-09-17T23:12:12.314 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2900 IOs, 1215174656 bytes 2024-09-17T23:12:12.391 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3000 IOs, 1230342144 bytes 2024-09-17T23:12:12.497 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3100 IOs, 1274778624 bytes 2024-09-17T23:12:12.656 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3200 IOs, 1381349376 bytes 2024-09-17T23:12:12.808 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3300 IOs, 1476439040 bytes 2024-09-17T23:12:12.917 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3400 IOs, 1539971072 bytes 2024-09-17T23:12:13.018 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3500 IOs, 1564506112 bytes 2024-09-17T23:12:13.115 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3600 IOs, 1596356608 bytes 2024-09-17T23:12:13.211 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3700 IOs, 1628660736 bytes 2024-09-17T23:12:13.327 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3800 IOs, 1680843776 bytes 2024-09-17T23:12:13.408 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3900 IOs, 1695723520 bytes 2024-09-17T23:12:13.503 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4000 IOs, 1728811008 bytes 2024-09-17T23:12:13.570 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4100 IOs, 1729432576 bytes 2024-09-17T23:12:13.661 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4200 IOs, 1753970688 bytes 2024-09-17T23:12:13.739 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4300 IOs, 1765063680 bytes 2024-09-17T23:12:13.815 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4400 IOs, 1774918656 bytes 2024-09-17T23:12:13.913 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4500 IOs, 1807176704 bytes 2024-09-17T23:12:14.067 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4600 IOs, 1905614848 bytes 2024-09-17T23:12:14.151 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4700 IOs, 1919503360 bytes 2024-09-17T23:12:14.253 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4800 IOs, 1956043776 bytes 2024-09-17T23:12:14.384 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4900 IOs, 2030370816 bytes 2024-09-17T23:12:14.481 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5000 IOs, 2057639936 bytes 2024-09-17T23:12:14.580 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5100 IOs, 2094163968 bytes 2024-09-17T23:12:14.701 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5200 IOs, 2151472128 bytes 2024-09-17T23:12:14.798 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5300 IOs, 2188780544 bytes 2024-09-17T23:12:14.881 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5400 IOs, 2211531776 bytes 2024-09-17T23:12:14.974 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5500 IOs, 2241989632 bytes 2024-09-17T23:12:15.079 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5600 IOs, 2285485056 bytes 2024-09-17T23:12:15.175 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5700 IOs, 2323923968 bytes 2024-09-17T23:12:15.282 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5800 IOs, 2364516352 bytes 2024-09-17T23:12:15.370 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5900 IOs, 2389773312 bytes 2024-09-17T23:12:15.501 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6000 IOs, 2457638912 bytes 2024-09-17T23:12:15.593 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6100 IOs, 2493873152 bytes 2024-09-17T23:12:15.693 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6200 IOs, 2524204032 bytes 2024-09-17T23:12:15.865 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6300 IOs, 2640651264 bytes 2024-09-17T23:12:15.957 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6400 IOs, 2668558336 bytes 2024-09-17T23:12:16.030 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6500 IOs, 2673042432 bytes 2024-09-17T23:12:16.042 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ error_code=124 2024-09-17T23:12:16.042 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set -e 2024-09-17T23:12:16.042 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' 124 -eq 124 ']' 2024-09-17T23:12:16.042 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:12:16.043 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-17T23:12:16.043 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:12:16.043 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:12:16.043 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:12:16.043 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state_pattern=up+replaying 2024-09-17T23:12:16.043 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local description_pattern= 2024-09-17T23:12:16.043 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local service_pattern= 2024-09-17T23:12:16.043 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:12:16.043 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:12:16.900 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.5760457637257607) > weight (0.25) 2024-09-17T23:12:16.900 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2024-09-17T23:12:16.900 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2024-09-17T23:12:16.900 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2024-09-17T23:12:16.901 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi038.stderr:2024-09-17T23:12:16.899+0000 7fedeec5d640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 36710) UID: 1000 2024-09-17T23:12:17.043 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-17T23:12:17.043 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:12:17.043 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:12:17.043 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:12:17.043 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state_pattern=up+replaying 2024-09-17T23:12:17.043 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local description_pattern= 2024-09-17T23:12:17.043 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local service_pattern= 2024-09-17T23:12:17.043 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status 2024-09-17T23:12:17.043 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ CEPH_ARGS= 2024-09-17T23:12:17.044 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-17T23:12:17.099 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ status='test: 2024-09-17T23:12:17.100 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: global_id: d6dfe272-2e04-4592-a96a-96648936ad18 2024-09-17T23:12:17.100 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: state: up+replaying 2024-09-17T23:12:17.100 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":530409472.0,"last_snapshot_bytes":525130752,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726614668,"remote_snapshot_timestamp":1726614668,"replay_state":"idle"} 2024-09-17T23:12:17.100 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: service: mirror.3 on smithi038 2024-09-17T23:12:17.100 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: last_update: 2024-09-17 23:11:55 2024-09-17T23:12:17.100 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: peer_sites: 2024-09-17T23:12:17.100 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: name: cluster2 2024-09-17T23:12:17.100 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: state: down+stopped 2024-09-17T23:12:17.100 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: description: local image is primary 2024-09-17T23:12:17.100 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: last_update: 2024-09-17 23:11:54' 2024-09-17T23:12:17.100 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep '^ state: .*up+replaying' 2024-09-17T23:12:17.101 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: state: up+replaying 2024-09-17T23:12:17.101 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep '^ description: .*' 2024-09-17T23:12:17.101 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":530409472.0,"last_snapshot_bytes":525130752,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726614668,"remote_snapshot_timestamp":1726614668,"replay_state":"idle"} 2024-09-17T23:12:17.102 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n '' ']' 2024-09-17T23:12:17.102 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ echo up+replaying 2024-09-17T23:12:17.102 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep '^up+' 2024-09-17T23:12:17.103 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:up+replaying 2024-09-17T23:12:17.103 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:12:17.104 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: service: mirror.3 on smithi038 2024-09-17T23:12:17.104 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local last_update 2024-09-17T23:12:17.105 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:12:17.106 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ last_update='2024-09-17 23:11:55' 2024-09-17T23:12:17.107 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-17 23:11:55' 2024-09-17T23:12:17.107 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:12:17.107 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:12:17.107 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:12:17.107 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state_pattern=up+replaying 2024-09-17T23:12:17.107 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local 'prev_last_update=2024-09-17 23:11:55' 2024-09-17T23:12:17.107 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status 2024-09-17T23:12:17.107 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ CEPH_ARGS= 2024-09-17T23:12:17.107 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-17T23:12:17.167 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ status='WARNINGOKWARNING142214298mirror.2smithi03819.1.1-330-gc2ddf972falseOK42234294mirror.3smithi03819.1.1-330-gc2ddf972trueOK42624310mirror.1smithi03819.1.1-330-gc2ddf972falseOK44834486mirror.0smithi03819.1.1-330-gc2ddf972falseOKtestd6dfe272-2e04-4592-a96a-96648936ad18up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":530409472.0,"last_snapshot_bytes":525130752,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726614668,"remote_snapshot_timestamp":1726614668,"replay_state":"idle"}42234294mirror.3smithi0382024-09-17 23:11:55cluster20ba5654d-47fb-4672-b20c-7359a1340314down+stoppedlocal image is primary2024-09-17 23:11:54' 2024-09-17T23:12:17.167 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local last_update state 2024-09-17T23:12:17.167 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-17T23:12:17.169 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ last_update='2024-09-17 23:11:55' 2024-09-17T23:12:17.169 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-17T23:12:17.171 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ state=up+replaying 2024-09-17T23:12:17.171 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ echo up+replaying 2024-09-17T23:12:17.171 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep up+replaying 2024-09-17T23:12:17.172 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:up+replaying 2024-09-17T23:12:17.172 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:12:17.173 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:12:17.173 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ snap_name=snap7 2024-09-17T23:12:17.173 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_snap cluster2 mirror test snap7 2024-09-17T23:12:17.173 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:12:17.173 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:12:17.173 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:12:17.173 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap7 2024-09-17T23:12:17.173 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap7 --debug-rbd=20 --debug-journaler=20 2024-09-17T23:12:24.901 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2024-09-17T23:12:24.902 INFO:teuthology.orchestra.run:waiting for 600 2024-09-17T23:12:24.902 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2024-09-17T23:12:24.902 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2024-09-17T23:12:24.903 DEBUG:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2024-09-17T23:12:24.906 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2024-09-17T23:12:24.906 INFO:tasks.rbd_mirror_thrash:waiting for 85 secs before thrashing 2024-09-17T23:12:46.122 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-17T23:12:46.122 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:12:46.122 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:12:46.122 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:12:46.122 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-17T23:12:46.122 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:12:46.122 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:12:46.122 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:12:46.122 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=started 2024-09-17T23:12:46.122 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:12:46.123 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:12:46.123 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:12:47.123 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-17T23:12:47.123 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:12:47.123 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:12:47.123 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:12:47.123 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=started 2024-09-17T23:12:47.123 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_result 2024-09-17T23:12:47.123 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=stopped 2024-09-17T23:12:47.124 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-17T23:12:47.124 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_instance=cluster1 2024-09-17T23:12:47.125 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:12:47.125 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ grep -i state 2024-09-17T23:12:47.125 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:12:47.125 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance=cluster1 2024-09-17T23:12:47.125 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local loop_instance 2024-09-17T23:12:47.125 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:12:47.125 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sleep 0 2024-09-17T23:12:47.125 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-17T23:12:47.127 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ seq 0 3 2024-09-17T23:12:47.127 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:12:47.127 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-17T23:12:47.127 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:0 2024-09-17T23:12:47.128 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:12:47.128 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:12:47.128 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-17T23:12:47.128 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:0 2024-09-17T23:12:47.128 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:12:47.128 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:12:47.128 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:12:47.128 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=0 2024-09-17T23:12:47.128 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-17T23:12:47.128 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:12:47.128 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:12:47.128 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=0 2024-09-17T23:12:47.128 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:12:47.129 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-17T23:12:47.129 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:0 2024-09-17T23:12:47.129 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:12:47.129 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:12:47.129 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-17T23:12:47.129 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:0 2024-09-17T23:12:47.129 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:12:47.129 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:12:47.129 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:12:47.129 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:12:47.129 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-17T23:12:47.129 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:12:47.130 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:12:47.130 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=0 2024-09-17T23:12:47.131 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=0 2024-09-17T23:12:47.131 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-17T23:12:47.150 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:12:47.150 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:12:47.150 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:12:47.150 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-17T23:12:47.232 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-17T23:12:47.233 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd mirror status : get status for rbd mirror 2024-09-17T23:12:47.233 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:admin_socket: invalid command 2024-09-17T23:12:47.235 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:12:47.235 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-17T23:12:47.236 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:1 2024-09-17T23:12:47.236 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:12:47.236 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:12:47.236 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-17T23:12:47.236 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:1 2024-09-17T23:12:47.236 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:12:47.236 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:12:47.236 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:12:47.236 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=1 2024-09-17T23:12:47.236 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-17T23:12:47.236 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:12:47.236 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:12:47.236 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=1 2024-09-17T23:12:47.237 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=1 2024-09-17T23:12:47.237 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-17T23:12:47.237 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:1 2024-09-17T23:12:47.237 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:12:47.237 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:12:47.237 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-17T23:12:47.237 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:1 2024-09-17T23:12:47.237 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:12:47.237 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:12:47.237 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:12:47.237 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=1 2024-09-17T23:12:47.237 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-17T23:12:47.237 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:12:47.237 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:12:47.238 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=1 2024-09-17T23:12:47.238 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=1 2024-09-17T23:12:47.238 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-17T23:12:47.256 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-17T23:12:47.256 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-17T23:12:47.256 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-17T23:12:47.256 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-17T23:12:47.332 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-17T23:12:47.332 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd mirror status : get status for rbd mirror 2024-09-17T23:12:47.332 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:admin_socket: invalid command 2024-09-17T23:12:47.335 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:12:47.335 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-17T23:12:47.335 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:2 2024-09-17T23:12:47.335 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:12:47.335 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:12:47.335 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-17T23:12:47.335 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:2 2024-09-17T23:12:47.335 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:12:47.336 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:12:47.336 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:12:47.336 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=2 2024-09-17T23:12:47.336 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-17T23:12:47.336 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:12:47.336 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:12:47.336 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=2 2024-09-17T23:12:47.336 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=2 2024-09-17T23:12:47.336 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-17T23:12:47.336 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:2 2024-09-17T23:12:47.336 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:12:47.336 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:12:47.337 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-17T23:12:47.337 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:2 2024-09-17T23:12:47.337 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:12:47.337 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:12:47.337 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:12:47.337 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=2 2024-09-17T23:12:47.337 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-17T23:12:47.337 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:12:47.337 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:12:47.337 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=2 2024-09-17T23:12:47.337 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=2 2024-09-17T23:12:47.337 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-17T23:12:47.354 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-17T23:12:47.354 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-17T23:12:47.354 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-17T23:12:47.354 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-17T23:12:47.430 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-17T23:12:47.430 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd mirror status : get status for rbd mirror 2024-09-17T23:12:47.430 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:admin_socket: invalid command 2024-09-17T23:12:47.433 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:12:47.433 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-09-17T23:12:47.433 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:3 2024-09-17T23:12:47.433 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:12:47.433 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:12:47.433 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-17T23:12:47.433 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:3 2024-09-17T23:12:47.433 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:12:47.433 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:12:47.433 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:12:47.433 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=3 2024-09-17T23:12:47.433 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-17T23:12:47.434 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:12:47.434 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:12:47.434 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=3 2024-09-17T23:12:47.434 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=3 2024-09-17T23:12:47.434 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-17T23:12:47.434 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:3 2024-09-17T23:12:47.434 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:12:47.434 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:12:47.434 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-17T23:12:47.434 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:3 2024-09-17T23:12:47.434 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:12:47.434 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:12:47.434 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:12:47.434 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=3 2024-09-17T23:12:47.435 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-17T23:12:47.435 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:12:47.435 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:12:47.435 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=3 2024-09-17T23:12:47.435 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=3 2024-09-17T23:12:47.435 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-17T23:12:47.451 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-17T23:12:47.451 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-17T23:12:47.451 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-17T23:12:47.452 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-09-17T23:12:47.535 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ return 0 2024-09-17T23:12:47.535 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ status_result=' "state": "Replaying"' 2024-09-17T23:12:47.535 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ echo ' "state": "Replaying"' 2024-09-17T23:12:47.536 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep -i Replaying 2024-09-17T23:12:47.536 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: "state": "Replaying" 2024-09-17T23:12:47.538 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_state=started 2024-09-17T23:12:47.538 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test started = started 2024-09-17T23:12:47.538 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:12:47.538 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-17T23:12:47.538 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:12:47.538 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:12:47.538 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:12:47.538 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:12:47.538 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = journal ']' 2024-09-17T23:12:47.538 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = snapshot ']' 2024-09-17T23:12:47.538 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-09-17T23:12:47.538 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:12:47.539 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:12:47.539 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:12:47.539 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:12:47.539 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster2-mirror-test.status 2024-09-17T23:12:47.539 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster2-mirror-test.status 2024-09-17T23:12:47.539 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:12:47.539 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-test.status 2024-09-17T23:12:47.540 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster1-mirror-test.status 2024-09-17T23:12:47.540 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster1-mirror-test.status 2024-09-17T23:12:47.540 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:12:47.540 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_status_log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-test.status 2024-09-17T23:12:47.541 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-17T23:12:47.541 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:12:47.541 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:12:47.541 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:12:47.541 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-17T23:12:49.131 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Snapshot ID: 23 2024-09-17T23:12:49.155 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.status 2024-09-17T23:12:49.156 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:12:49.156 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:12:49.156 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:12:49.156 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-test.status 2024-09-17T23:12:49.156 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-17T23:12:49.156 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:12:49.235 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-17T23:12:49.237 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snapshot_id=23 2024-09-17T23:12:49.237 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:12:49.237 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:12:49.237 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.2 2024-09-17T23:12:49.438 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.status 2024-09-17T23:12:49.438 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:12:49.438 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:12:49.438 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:12:49.438 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-test.status 2024-09-17T23:12:49.438 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-17T23:12:49.438 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:12:49.490 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:12:49.491 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:12:49.492 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:12:49.492 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:12:49.492 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:12:49.492 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:12:49.492 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 23 = '' 2024-09-17T23:12:49.493 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:12:49.493 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.4 2024-09-17T23:12:49.893 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.status 2024-09-17T23:12:49.894 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:12:49.894 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:12:49.894 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:12:49.894 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-test.status 2024-09-17T23:12:49.894 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-17T23:12:49.894 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:12:49.947 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:12:49.947 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:12:49.949 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:12:49.949 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:12:49.949 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:12:49.950 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:12:49.950 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 23 = '' 2024-09-17T23:12:49.950 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:12:49.950 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.8 2024-09-17T23:12:50.751 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.status 2024-09-17T23:12:50.751 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:12:50.751 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:12:50.752 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:12:50.752 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-test.status 2024-09-17T23:12:50.752 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-17T23:12:50.752 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:12:50.806 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:12:50.807 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:12:50.808 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:12:50.808 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:12:50.808 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:12:50.809 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:12:50.809 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 23 = '' 2024-09-17T23:12:50.809 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:12:50.809 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1.6 2024-09-17T23:12:52.410 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.status 2024-09-17T23:12:52.410 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:12:52.410 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:12:52.410 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:12:52.410 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-test.status 2024-09-17T23:12:52.411 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-17T23:12:52.411 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:12:52.464 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:12:52.466 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id=23 2024-09-17T23:12:52.466 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 23 = 23 2024-09-17T23:12:52.466 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:12:52.466 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snap_present cluster1 mirror test snap7 2024-09-17T23:12:52.467 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:12:52.467 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:12:52.467 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:12:52.467 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap7 2024-09-17T23:12:52.467 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:12:52.467 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-17T23:12:52.467 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:12:53.467 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 info mirror/test@snap7 2024-09-17T23:12:53.518 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'test': 2024-09-17T23:12:53.518 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 512 MiB in 128 objects 2024-09-17T23:12:53.518 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:12:53.518 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 14 2024-09-17T23:12:53.518 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 10c7e03f2edd 2024-09-17T23:12:53.519 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.10c7e03f2edd 2024-09-17T23:12:53.519 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:12:53.519 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-17T23:12:53.519 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:12:53.519 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:12:53.519 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:06:57 2024 2024-09-17T23:12:53.519 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:12:06 2024 2024-09-17T23:12:53.519 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:06:57 2024 2024-09-17T23:12:53.519 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: protected: False 2024-09-17T23:12:53.519 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:12:53.519 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:12:53.520 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: d6dfe272-2e04-4592-a96a-96648936ad18 2024-09-17T23:12:53.520 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: false 2024-09-17T23:12:53.522 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:12:53.522 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n snap6-clean ']' 2024-09-17T23:12:53.522 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ compare_image_snaps mirror test snap6-clean 2024-09-17T23:12:53.522 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:12:53.522 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:12:53.522 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap6-clean 2024-09-17T23:12:53.523 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local ret=0 2024-09-17T23:12:53.523 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local rmt_export=/tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export 2024-09-17T23:12:53.523 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local loc_export=/tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:12:53.523 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:12:53.523 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6-clean /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export 2024-09-17T23:12:54.049 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-17T23:12:54.057 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6-clean /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:12:54.749 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-17T23:12:54.757 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ cmp /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:12:55.005 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:12:55.123 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:12:55.123 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ compare_image_snaps mirror test snap7 2024-09-17T23:12:55.123 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:12:55.123 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:12:55.123 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap7 2024-09-17T23:12:55.123 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local ret=0 2024-09-17T23:12:55.123 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local rmt_export=/tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export 2024-09-17T23:12:55.123 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local loc_export=/tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:12:55.123 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:12:55.124 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7 /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export 2024-09-17T23:12:55.654 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-17T23:12:55.662 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7 /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:12:56.361 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-17T23:12:56.368 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ cmp /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:12:56.619 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:12:56.739 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:12:56.739 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ clean_snap_name=snap7-clean 2024-09-17T23:12:56.739 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_snap cluster2 mirror test snap7-clean 2024-09-17T23:12:56.739 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:12:56.739 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:12:56.739 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:12:56.739 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap7-clean 2024-09-17T23:12:56.739 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap7-clean --debug-rbd=20 --debug-journaler=20 2024-09-17T23:12:57.038 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 10` 2024-09-17T23:12:57.038 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ stress_write_image cluster2 mirror test 2024-09-17T23:12:57.038 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:12:57.038 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:12:57.038 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:12:57.039 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-17T23:12:57.040 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local duration=6 2024-09-17T23:12:57.040 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set +e 2024-09-17T23:12:57.040 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-17T23:12:57.314 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:100 IOs, 31095808 bytes 2024-09-17T23:12:57.458 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:200 IOs, 50760704 bytes 2024-09-17T23:12:57.629 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:300 IOs, 124386304 bytes 2024-09-17T23:12:57.739 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:400 IOs, 157188096 bytes 2024-09-17T23:12:57.847 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:500 IOs, 195329024 bytes 2024-09-17T23:12:57.998 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:600 IOs, 283916288 bytes 2024-09-17T23:12:58.116 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:700 IOs, 352706560 bytes 2024-09-17T23:12:58.232 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:800 IOs, 380253184 bytes 2024-09-17T23:12:58.337 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:900 IOs, 443783168 bytes 2024-09-17T23:12:58.466 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1000 IOs, 490036224 bytes 2024-09-17T23:12:58.598 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1100 IOs, 561958912 bytes 2024-09-17T23:12:58.740 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1200 IOs, 639163392 bytes 2024-09-17T23:12:58.808 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1300 IOs, 640724992 bytes 2024-09-17T23:12:58.917 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1400 IOs, 692030464 bytes 2024-09-17T23:12:59.006 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1500 IOs, 714711040 bytes 2024-09-17T23:12:59.097 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1600 IOs, 741914624 bytes 2024-09-17T23:12:59.192 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1700 IOs, 774536192 bytes 2024-09-17T23:12:59.281 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1800 IOs, 790817792 bytes 2024-09-17T23:12:59.384 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1900 IOs, 832716800 bytes 2024-09-17T23:12:59.467 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2000 IOs, 852911104 bytes 2024-09-17T23:12:59.548 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2100 IOs, 866087936 bytes 2024-09-17T23:12:59.709 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2200 IOs, 974595072 bytes 2024-09-17T23:12:59.814 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2300 IOs, 1024876544 bytes 2024-09-17T23:12:59.915 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2400 IOs, 1039198208 bytes 2024-09-17T23:13:00.013 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2500 IOs, 1072037888 bytes 2024-09-17T23:13:00.104 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2600 IOs, 1096704000 bytes 2024-09-17T23:13:00.212 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2700 IOs, 1140935680 bytes 2024-09-17T23:13:00.303 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2800 IOs, 1167155200 bytes 2024-09-17T23:13:00.400 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2900 IOs, 1198849024 bytes 2024-09-17T23:13:00.491 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3000 IOs, 1221211136 bytes 2024-09-17T23:13:00.590 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3100 IOs, 1251382272 bytes 2024-09-17T23:13:00.680 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3200 IOs, 1274229760 bytes 2024-09-17T23:13:00.852 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3300 IOs, 1395820544 bytes 2024-09-17T23:13:00.997 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3400 IOs, 1461719040 bytes 2024-09-17T23:13:01.106 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3500 IOs, 1510498304 bytes 2024-09-17T23:13:01.227 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3600 IOs, 1567758336 bytes 2024-09-17T23:13:01.327 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3700 IOs, 1599830016 bytes 2024-09-17T23:13:01.421 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3800 IOs, 1628037120 bytes 2024-09-17T23:13:01.503 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3900 IOs, 1642598400 bytes 2024-09-17T23:13:01.640 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4000 IOs, 1716247552 bytes 2024-09-17T23:13:01.771 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4100 IOs, 1803181056 bytes 2024-09-17T23:13:01.859 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4200 IOs, 1803791360 bytes 2024-09-17T23:13:02.003 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4300 IOs, 1883497472 bytes 2024-09-17T23:13:02.116 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4400 IOs, 1942681600 bytes 2024-09-17T23:13:02.224 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4500 IOs, 1980547072 bytes 2024-09-17T23:13:02.310 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4600 IOs, 2000524288 bytes 2024-09-17T23:13:02.396 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4700 IOs, 2020459520 bytes 2024-09-17T23:13:02.512 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4800 IOs, 2072013824 bytes 2024-09-17T23:13:02.600 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4900 IOs, 2095734784 bytes 2024-09-17T23:13:02.718 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5000 IOs, 2150927360 bytes 2024-09-17T23:13:02.857 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5100 IOs, 2226939904 bytes 2024-09-17T23:13:02.946 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5200 IOs, 2250568704 bytes 2024-09-17T23:13:03.032 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5300 IOs, 2269112320 bytes 2024-09-17T23:13:03.044 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ error_code=124 2024-09-17T23:13:03.045 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set -e 2024-09-17T23:13:03.045 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' 124 -eq 124 ']' 2024-09-17T23:13:03.045 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:13:03.045 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-17T23:13:03.045 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:13:03.045 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:13:03.045 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:13:03.045 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state_pattern=up+replaying 2024-09-17T23:13:03.045 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local description_pattern= 2024-09-17T23:13:03.045 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local service_pattern= 2024-09-17T23:13:03.045 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:13:03.045 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:13:04.046 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-17T23:13:04.046 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:13:04.046 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:13:04.046 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:13:04.047 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state_pattern=up+replaying 2024-09-17T23:13:04.047 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local description_pattern= 2024-09-17T23:13:04.047 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local service_pattern= 2024-09-17T23:13:04.047 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status 2024-09-17T23:13:04.047 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ CEPH_ARGS= 2024-09-17T23:13:04.047 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-17T23:13:04.109 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ status='test: 2024-09-17T23:13:04.109 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: global_id: d6dfe272-2e04-4592-a96a-96648936ad18 2024-09-17T23:13:04.109 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: state: up+replaying 2024-09-17T23:13:04.109 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: description: replaying, {"bytes_per_second":17765580.8,"bytes_per_snapshot":534520320.0,"last_snapshot_bytes":532967424,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726614768,"remote_snapshot_timestamp":1726614768,"replay_state":"idle"} 2024-09-17T23:13:04.110 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: service: mirror.3 on smithi038 2024-09-17T23:13:04.110 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: last_update: 2024-09-17 23:12:55 2024-09-17T23:13:04.110 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: peer_sites: 2024-09-17T23:13:04.110 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: name: cluster2 2024-09-17T23:13:04.110 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: state: up+stopped 2024-09-17T23:13:04.110 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: description: local image is primary 2024-09-17T23:13:04.110 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: last_update: 2024-09-17 23:12:56' 2024-09-17T23:13:04.110 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep '^ state: .*up+replaying' 2024-09-17T23:13:04.110 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: state: up+replaying 2024-09-17T23:13:04.111 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep '^ description: .*' 2024-09-17T23:13:04.112 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: description: replaying, {"bytes_per_second":17765580.8,"bytes_per_snapshot":534520320.0,"last_snapshot_bytes":532967424,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726614768,"remote_snapshot_timestamp":1726614768,"replay_state":"idle"} 2024-09-17T23:13:04.112 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n '' ']' 2024-09-17T23:13:04.112 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ echo up+replaying 2024-09-17T23:13:04.112 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep '^up+' 2024-09-17T23:13:04.113 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:up+replaying 2024-09-17T23:13:04.113 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:13:04.114 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: service: mirror.3 on smithi038 2024-09-17T23:13:04.115 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local last_update 2024-09-17T23:13:04.115 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:13:04.117 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ last_update='2024-09-17 23:12:55' 2024-09-17T23:13:04.117 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-17 23:12:55' 2024-09-17T23:13:04.117 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:13:04.117 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:13:04.118 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:13:04.118 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state_pattern=up+replaying 2024-09-17T23:13:04.118 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local 'prev_last_update=2024-09-17 23:12:55' 2024-09-17T23:13:04.118 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status 2024-09-17T23:13:04.118 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ CEPH_ARGS= 2024-09-17T23:13:04.118 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-17T23:13:04.180 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ status='OKOKOK142214298mirror.2smithi03819.1.1-330-gc2ddf972falseOK42234294mirror.3smithi03819.1.1-330-gc2ddf972trueOK42624310mirror.1smithi03819.1.1-330-gc2ddf972falseOK44834486mirror.0smithi03819.1.1-330-gc2ddf972falseOKtestd6dfe272-2e04-4592-a96a-96648936ad18up+replayingreplaying, {"bytes_per_second":17765580.8,"bytes_per_snapshot":534520320.0,"last_snapshot_bytes":532967424,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726614768,"remote_snapshot_timestamp":1726614768,"replay_state":"idle"}42234294mirror.3smithi0382024-09-17 23:12:55cluster20ba5654d-47fb-4672-b20c-7359a1340314up+stoppedlocal image is primary2024-09-17 23:12:56' 2024-09-17T23:13:04.180 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local last_update state 2024-09-17T23:13:04.180 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-17T23:13:04.182 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ last_update='2024-09-17 23:12:55' 2024-09-17T23:13:04.182 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-17T23:13:04.184 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ state=up+replaying 2024-09-17T23:13:04.184 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ echo up+replaying 2024-09-17T23:13:04.184 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep up+replaying 2024-09-17T23:13:04.186 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:up+replaying 2024-09-17T23:13:04.186 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:13:04.187 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:13:04.187 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ snap_name=snap8 2024-09-17T23:13:04.187 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_snap cluster2 mirror test snap8 2024-09-17T23:13:04.187 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:13:04.187 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:13:04.187 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:13:04.187 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap8 2024-09-17T23:13:04.187 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap8 --debug-rbd=20 --debug-journaler=20 2024-09-17T23:13:09.798 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-09-17T23:13:09.799 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-09-17T23:13:09.799 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2024-09-17T23:13:09.800 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi038.stderr:2024-09-17T23:13:09.798+0000 7f70e6cf2640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 41116) UID: 1000 2024-09-17T23:13:17.800 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-09-17T23:13:17.801 INFO:teuthology.orchestra.run:waiting for 600 2024-09-17T23:13:17.801 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-09-17T23:13:17.801 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-09-17T23:13:17.802 DEBUG:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-09-17T23:13:17.805 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-09-17T23:13:17.805 INFO:tasks.rbd_mirror_thrash:waiting for 97 secs before thrashing 2024-09-17T23:13:33.164 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-17T23:13:33.164 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:13:33.164 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:13:33.164 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:13:33.165 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-17T23:13:33.165 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:13:33.165 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:13:33.165 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:13:33.165 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=started 2024-09-17T23:13:33.165 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:13:33.165 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:13:33.165 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:13:34.165 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-17T23:13:34.166 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:13:34.166 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:13:34.166 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:13:34.166 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=started 2024-09-17T23:13:34.166 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_result 2024-09-17T23:13:34.166 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=stopped 2024-09-17T23:13:34.167 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-17T23:13:34.167 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_instance=cluster1 2024-09-17T23:13:34.167 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ grep -i state 2024-09-17T23:13:34.167 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:13:34.167 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:13:34.167 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance=cluster1 2024-09-17T23:13:34.167 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local loop_instance 2024-09-17T23:13:34.167 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:13:34.168 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sleep 0 2024-09-17T23:13:34.168 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-17T23:13:34.169 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ seq 0 3 2024-09-17T23:13:34.169 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:13:34.169 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-17T23:13:34.170 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:0 2024-09-17T23:13:34.170 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:13:34.170 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:13:34.170 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-17T23:13:34.170 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:0 2024-09-17T23:13:34.170 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:13:34.170 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:13:34.170 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:13:34.170 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=0 2024-09-17T23:13:34.170 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-17T23:13:34.170 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:13:34.170 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:13:34.170 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=0 2024-09-17T23:13:34.170 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:13:34.171 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-17T23:13:34.172 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:0 2024-09-17T23:13:34.172 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:13:34.172 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:13:34.172 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-17T23:13:34.172 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:0 2024-09-17T23:13:34.172 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:13:34.172 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:13:34.172 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:13:34.172 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:13:34.173 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-17T23:13:34.173 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:13:34.173 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:13:34.173 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=0 2024-09-17T23:13:34.173 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=0 2024-09-17T23:13:34.173 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-17T23:13:34.194 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:13:34.194 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:13:34.194 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:13:34.194 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-17T23:13:34.276 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-17T23:13:34.276 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd mirror status : get status for rbd mirror 2024-09-17T23:13:34.276 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:admin_socket: invalid command 2024-09-17T23:13:34.279 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:13:34.279 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-17T23:13:34.280 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:1 2024-09-17T23:13:34.280 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:13:34.280 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:13:34.280 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-17T23:13:34.280 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:1 2024-09-17T23:13:34.280 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:13:34.280 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:13:34.280 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:13:34.280 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=1 2024-09-17T23:13:34.280 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-17T23:13:34.280 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:13:34.280 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:13:34.280 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=1 2024-09-17T23:13:34.281 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=1 2024-09-17T23:13:34.281 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-17T23:13:34.281 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:1 2024-09-17T23:13:34.281 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:13:34.281 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:13:34.281 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-17T23:13:34.281 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:1 2024-09-17T23:13:34.281 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:13:34.281 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:13:34.281 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:13:34.281 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=1 2024-09-17T23:13:34.281 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-17T23:13:34.281 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:13:34.281 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:13:34.282 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=1 2024-09-17T23:13:34.282 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=1 2024-09-17T23:13:34.282 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-17T23:13:34.299 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-17T23:13:34.299 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-17T23:13:34.299 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-17T23:13:34.299 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-17T23:13:34.375 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-17T23:13:34.375 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd mirror status : get status for rbd mirror 2024-09-17T23:13:34.375 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:admin_socket: invalid command 2024-09-17T23:13:34.378 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:13:34.378 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-17T23:13:34.378 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:2 2024-09-17T23:13:34.378 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:13:34.378 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:13:34.378 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-17T23:13:34.378 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:2 2024-09-17T23:13:34.378 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:13:34.378 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:13:34.378 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:13:34.378 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=2 2024-09-17T23:13:34.378 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-17T23:13:34.378 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:13:34.379 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:13:34.379 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=2 2024-09-17T23:13:34.379 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=2 2024-09-17T23:13:34.379 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-17T23:13:34.379 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:2 2024-09-17T23:13:34.379 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:13:34.379 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:13:34.379 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-17T23:13:34.379 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:2 2024-09-17T23:13:34.379 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:13:34.379 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:13:34.379 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:13:34.379 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=2 2024-09-17T23:13:34.379 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-17T23:13:34.380 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:13:34.380 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:13:34.380 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=2 2024-09-17T23:13:34.380 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=2 2024-09-17T23:13:34.380 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-17T23:13:34.396 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-17T23:13:34.396 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-17T23:13:34.396 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-17T23:13:34.396 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-17T23:13:34.472 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-17T23:13:34.472 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd mirror status : get status for rbd mirror 2024-09-17T23:13:34.472 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:admin_socket: invalid command 2024-09-17T23:13:34.474 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:13:34.474 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-09-17T23:13:34.475 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:3 2024-09-17T23:13:34.475 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:13:34.475 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:13:34.475 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-17T23:13:34.475 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:3 2024-09-17T23:13:34.475 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:13:34.475 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:13:34.475 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:13:34.475 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=3 2024-09-17T23:13:34.475 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-17T23:13:34.475 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:13:34.475 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:13:34.475 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=3 2024-09-17T23:13:34.476 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=3 2024-09-17T23:13:34.476 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-17T23:13:34.476 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:3 2024-09-17T23:13:34.476 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:13:34.476 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:13:34.476 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-17T23:13:34.476 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:3 2024-09-17T23:13:34.476 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:13:34.476 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:13:34.476 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:13:34.476 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=3 2024-09-17T23:13:34.477 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-17T23:13:34.477 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:13:34.477 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:13:34.477 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=3 2024-09-17T23:13:34.477 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=3 2024-09-17T23:13:34.477 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-17T23:13:34.493 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-17T23:13:34.494 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-17T23:13:34.494 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-17T23:13:34.494 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-09-17T23:13:34.578 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ return 0 2024-09-17T23:13:34.578 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ status_result=' "state": "Replaying"' 2024-09-17T23:13:34.578 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ echo ' "state": "Replaying"' 2024-09-17T23:13:34.579 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep -i Replaying 2024-09-17T23:13:34.579 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: "state": "Replaying" 2024-09-17T23:13:34.580 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_state=started 2024-09-17T23:13:34.581 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test started = started 2024-09-17T23:13:34.581 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:13:34.581 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-17T23:13:34.581 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:13:34.581 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:13:34.581 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:13:34.581 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:13:34.581 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = journal ']' 2024-09-17T23:13:34.581 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = snapshot ']' 2024-09-17T23:13:34.581 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-09-17T23:13:34.581 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:13:34.581 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:13:34.582 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:13:34.582 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:13:34.582 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster2-mirror-test.status 2024-09-17T23:13:34.582 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster2-mirror-test.status 2024-09-17T23:13:34.582 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:13:34.583 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-test.status 2024-09-17T23:13:34.583 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster1-mirror-test.status 2024-09-17T23:13:34.583 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster1-mirror-test.status 2024-09-17T23:13:34.583 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:13:34.584 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_status_log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-test.status 2024-09-17T23:13:34.584 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-17T23:13:34.584 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:13:34.584 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:13:34.584 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:13:34.584 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-17T23:13:36.108 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Snapshot ID: 26 2024-09-17T23:13:36.130 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.status 2024-09-17T23:13:36.130 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:13:36.130 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:13:36.130 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:13:36.130 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-test.status 2024-09-17T23:13:36.131 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:13:36.131 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-17T23:13:36.212 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-17T23:13:36.215 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snapshot_id=26 2024-09-17T23:13:36.215 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:13:36.215 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:13:36.215 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.2 2024-09-17T23:13:36.417 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.status 2024-09-17T23:13:36.417 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:13:36.417 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:13:36.417 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:13:36.417 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-test.status 2024-09-17T23:13:36.417 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-17T23:13:36.417 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:13:36.468 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:13:36.468 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:13:36.469 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:13:36.469 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:13:36.470 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:13:36.470 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:13:36.470 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 26 = '' 2024-09-17T23:13:36.470 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:13:36.470 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.4 2024-09-17T23:13:36.872 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.status 2024-09-17T23:13:36.873 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:13:36.873 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:13:36.873 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:13:36.873 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-test.status 2024-09-17T23:13:36.873 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-17T23:13:36.873 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:13:36.928 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:13:36.928 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:13:36.929 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:13:36.929 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:13:36.930 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:13:36.931 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:13:36.931 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 26 = '' 2024-09-17T23:13:36.931 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:13:36.931 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.8 2024-09-17T23:13:37.732 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.status 2024-09-17T23:13:37.732 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:13:37.732 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:13:37.732 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:13:37.732 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-test.status 2024-09-17T23:13:37.732 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-17T23:13:37.732 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:13:37.790 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:13:37.790 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:13:37.791 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:13:37.791 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:13:37.791 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:13:37.792 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:13:37.792 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 26 = '' 2024-09-17T23:13:37.792 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:13:37.792 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1.6 2024-09-17T23:13:39.394 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.status 2024-09-17T23:13:39.394 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:13:39.394 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:13:39.394 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:13:39.394 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-test.status 2024-09-17T23:13:39.394 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-17T23:13:39.394 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:13:39.447 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:13:39.449 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id=26 2024-09-17T23:13:39.449 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 26 = 26 2024-09-17T23:13:39.449 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:13:39.449 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snap_present cluster1 mirror test snap8 2024-09-17T23:13:39.449 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:13:39.449 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:13:39.449 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:13:39.449 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap8 2024-09-17T23:13:39.449 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:13:39.450 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-17T23:13:39.450 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:13:40.450 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 info mirror/test@snap8 2024-09-17T23:13:40.500 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'test': 2024-09-17T23:13:40.500 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 512 MiB in 128 objects 2024-09-17T23:13:40.500 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:13:40.500 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 16 2024-09-17T23:13:40.500 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 10c7e03f2edd 2024-09-17T23:13:40.500 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.10c7e03f2edd 2024-09-17T23:13:40.500 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:13:40.500 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-17T23:13:40.500 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:13:40.501 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:13:40.501 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:06:57 2024 2024-09-17T23:13:40.501 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:12:06 2024 2024-09-17T23:13:40.501 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:06:57 2024 2024-09-17T23:13:40.501 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: protected: False 2024-09-17T23:13:40.501 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:13:40.501 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:13:40.501 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: d6dfe272-2e04-4592-a96a-96648936ad18 2024-09-17T23:13:40.501 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: false 2024-09-17T23:13:40.503 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:13:40.504 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n snap7-clean ']' 2024-09-17T23:13:40.504 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ compare_image_snaps mirror test snap7-clean 2024-09-17T23:13:40.504 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:13:40.504 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:13:40.504 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap7-clean 2024-09-17T23:13:40.504 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local ret=0 2024-09-17T23:13:40.504 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local rmt_export=/tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export 2024-09-17T23:13:40.504 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local loc_export=/tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:13:40.504 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:13:40.504 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7-clean /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export 2024-09-17T23:13:41.032 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-17T23:13:41.042 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7-clean /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:13:41.767 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-17T23:13:41.775 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ cmp /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:13:42.026 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:13:42.146 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:13:42.146 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ compare_image_snaps mirror test snap8 2024-09-17T23:13:42.146 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:13:42.146 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:13:42.146 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap8 2024-09-17T23:13:42.147 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local ret=0 2024-09-17T23:13:42.147 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local rmt_export=/tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export 2024-09-17T23:13:42.147 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local loc_export=/tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:13:42.147 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:13:42.147 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8 /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export 2024-09-17T23:13:42.655 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-17T23:13:42.665 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8 /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:13:43.418 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-17T23:13:43.427 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ cmp /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:13:43.672 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:13:43.791 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:13:43.791 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ clean_snap_name=snap8-clean 2024-09-17T23:13:43.792 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_snap cluster2 mirror test snap8-clean 2024-09-17T23:13:43.792 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:13:43.792 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:13:43.792 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:13:43.792 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap8-clean 2024-09-17T23:13:43.792 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap8-clean --debug-rbd=20 --debug-journaler=20 2024-09-17T23:13:44.057 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 10` 2024-09-17T23:13:44.057 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ stress_write_image cluster2 mirror test 2024-09-17T23:13:44.057 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:13:44.057 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:13:44.057 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:13:44.057 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-17T23:13:44.058 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local duration=5 2024-09-17T23:13:44.058 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set +e 2024-09-17T23:13:44.059 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-17T23:13:44.338 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:100 IOs, 30120960 bytes 2024-09-17T23:13:44.585 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:200 IOs, 130123776 bytes 2024-09-17T23:13:44.698 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:300 IOs, 159574016 bytes 2024-09-17T23:13:44.823 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:400 IOs, 188779520 bytes 2024-09-17T23:13:44.913 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:500 IOs, 210391040 bytes 2024-09-17T23:13:45.004 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:600 IOs, 235380736 bytes 2024-09-17T23:13:45.128 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:700 IOs, 296997888 bytes 2024-09-17T23:13:45.241 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:800 IOs, 343709696 bytes 2024-09-17T23:13:45.443 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:900 IOs, 489333760 bytes 2024-09-17T23:13:45.531 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1000 IOs, 502137856 bytes 2024-09-17T23:13:45.649 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1100 IOs, 548873216 bytes 2024-09-17T23:13:45.795 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1200 IOs, 628749312 bytes 2024-09-17T23:13:45.916 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1300 IOs, 684218368 bytes 2024-09-17T23:13:46.036 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1400 IOs, 732751872 bytes 2024-09-17T23:13:46.136 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1500 IOs, 763136000 bytes 2024-09-17T23:13:46.224 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1600 IOs, 782548992 bytes 2024-09-17T23:13:46.361 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1700 IOs, 855831552 bytes 2024-09-17T23:13:46.506 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1800 IOs, 935888896 bytes 2024-09-17T23:13:46.599 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1900 IOs, 969743360 bytes 2024-09-17T23:13:46.702 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2000 IOs, 993357824 bytes 2024-09-17T23:13:46.794 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2100 IOs, 1017107456 bytes 2024-09-17T23:13:46.885 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2200 IOs, 1039140864 bytes 2024-09-17T23:13:46.979 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2300 IOs, 1067546624 bytes 2024-09-17T23:13:47.083 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2400 IOs, 1103631360 bytes 2024-09-17T23:13:47.181 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2500 IOs, 1139558400 bytes 2024-09-17T23:13:47.284 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2600 IOs, 1171517440 bytes 2024-09-17T23:13:47.391 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2700 IOs, 1223737344 bytes 2024-09-17T23:13:47.499 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2800 IOs, 1255194624 bytes 2024-09-17T23:13:47.627 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2900 IOs, 1319516160 bytes 2024-09-17T23:13:47.759 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3000 IOs, 1385179136 bytes 2024-09-17T23:13:47.842 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3100 IOs, 1403956224 bytes 2024-09-17T23:13:47.919 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3200 IOs, 1411950592 bytes 2024-09-17T23:13:48.051 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3300 IOs, 1469749248 bytes 2024-09-17T23:13:48.170 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3400 IOs, 1523423232 bytes 2024-09-17T23:13:48.285 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3500 IOs, 1577829376 bytes 2024-09-17T23:13:48.392 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3600 IOs, 1613846528 bytes 2024-09-17T23:13:48.488 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3700 IOs, 1646088192 bytes 2024-09-17T23:13:48.591 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3800 IOs, 1679905792 bytes 2024-09-17T23:13:48.705 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3900 IOs, 1724492800 bytes 2024-09-17T23:13:48.792 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4000 IOs, 1746015232 bytes 2024-09-17T23:13:48.967 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4100 IOs, 1860553728 bytes 2024-09-17T23:13:49.065 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ error_code=124 2024-09-17T23:13:49.065 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set -e 2024-09-17T23:13:49.065 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' 124 -eq 124 ']' 2024-09-17T23:13:49.065 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:13:49.065 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-17T23:13:49.065 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:13:49.065 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:13:49.065 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:13:49.065 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state_pattern=up+replaying 2024-09-17T23:13:49.065 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local description_pattern= 2024-09-17T23:13:49.065 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local service_pattern= 2024-09-17T23:13:49.066 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:13:49.066 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:13:49.907 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.5529120820615613) > weight (0.25) 2024-09-17T23:13:49.907 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.43409353534777106) > weight (0.25) 2024-09-17T23:13:49.907 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.7821257736023015) > weight (0.25) 2024-09-17T23:13:49.907 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.7956267434292332) > weight (0.25) 2024-09-17T23:13:49.907 INFO:tasks.rbd_mirror_thrash:waiting for 109 secs before thrashing 2024-09-17T23:13:50.066 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-17T23:13:50.066 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:13:50.066 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:13:50.066 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:13:50.066 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state_pattern=up+replaying 2024-09-17T23:13:50.066 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local description_pattern= 2024-09-17T23:13:50.067 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local service_pattern= 2024-09-17T23:13:50.067 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status 2024-09-17T23:13:50.067 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ CEPH_ARGS= 2024-09-17T23:13:50.067 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-17T23:13:50.128 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ status='test: 2024-09-17T23:13:50.128 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: global_id: d6dfe272-2e04-4592-a96a-96648936ad18 2024-09-17T23:13:50.128 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: state: up+replaying 2024-09-17T23:13:50.128 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: description: replaying, {"bytes_per_second":17765580.8,"bytes_per_snapshot":534520320.0,"last_snapshot_bytes":532967424,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726614768,"remote_snapshot_timestamp":1726614768,"replay_state":"idle"} 2024-09-17T23:13:50.128 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: service: mirror.3 on smithi038 2024-09-17T23:13:50.128 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: last_update: 2024-09-17 23:13:25 2024-09-17T23:13:50.128 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: peer_sites: 2024-09-17T23:13:50.128 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: name: cluster2 2024-09-17T23:13:50.128 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: state: up+stopped 2024-09-17T23:13:50.128 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: description: local image is primary 2024-09-17T23:13:50.129 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: last_update: 2024-09-17 23:13:26' 2024-09-17T23:13:50.129 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep '^ state: .*up+replaying' 2024-09-17T23:13:50.129 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: state: up+replaying 2024-09-17T23:13:50.129 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep '^ description: .*' 2024-09-17T23:13:50.130 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: description: replaying, {"bytes_per_second":17765580.8,"bytes_per_snapshot":534520320.0,"last_snapshot_bytes":532967424,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726614768,"remote_snapshot_timestamp":1726614768,"replay_state":"idle"} 2024-09-17T23:13:50.130 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n '' ']' 2024-09-17T23:13:50.131 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ echo up+replaying 2024-09-17T23:13:50.131 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep '^up+' 2024-09-17T23:13:50.131 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:up+replaying 2024-09-17T23:13:50.132 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:13:50.133 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: service: mirror.3 on smithi038 2024-09-17T23:13:50.133 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local last_update 2024-09-17T23:13:50.134 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:13:50.135 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ last_update='2024-09-17 23:13:25' 2024-09-17T23:13:50.135 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-17 23:13:25' 2024-09-17T23:13:50.136 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:13:50.136 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:13:50.136 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:13:50.136 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state_pattern=up+replaying 2024-09-17T23:13:50.136 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local 'prev_last_update=2024-09-17 23:13:25' 2024-09-17T23:13:50.136 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status 2024-09-17T23:13:50.136 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ CEPH_ARGS= 2024-09-17T23:13:50.136 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-17T23:13:50.197 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ status='OKOKOK142214298mirror.2smithi03819.1.1-330-gc2ddf972falseOK42234294mirror.3smithi03819.1.1-330-gc2ddf972trueOK42624310mirror.1smithi03819.1.1-330-gc2ddf972falseOK45584561mirror.0smithi03819.1.1-330-gc2ddf972falseOKtestd6dfe272-2e04-4592-a96a-96648936ad18up+replayingreplaying, {"bytes_per_second":17765580.8,"bytes_per_snapshot":534520320.0,"last_snapshot_bytes":532967424,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726614768,"remote_snapshot_timestamp":1726614768,"replay_state":"idle"}42234294mirror.3smithi0382024-09-17 23:13:25cluster20ba5654d-47fb-4672-b20c-7359a1340314up+stoppedlocal image is primary2024-09-17 23:13:26' 2024-09-17T23:13:50.198 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local last_update state 2024-09-17T23:13:50.198 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-17T23:13:50.199 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ last_update='2024-09-17 23:13:25' 2024-09-17T23:13:50.200 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-17T23:13:50.201 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ state=up+replaying 2024-09-17T23:13:50.202 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ echo up+replaying 2024-09-17T23:13:50.202 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep up+replaying 2024-09-17T23:13:50.202 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:up+replaying 2024-09-17T23:13:50.203 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:13:50.203 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:13:50.203 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ snap_name=snap9 2024-09-17T23:13:50.204 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_snap cluster2 mirror test snap9 2024-09-17T23:13:50.204 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:13:50.204 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:13:50.204 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:13:50.204 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap9 2024-09-17T23:13:50.204 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap9 --debug-rbd=20 --debug-journaler=20 2024-09-17T23:14:19.771 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-17T23:14:19.772 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:14:19.772 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:14:19.772 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:14:19.772 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-17T23:14:19.772 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:14:19.772 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:14:19.772 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:14:19.772 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=started 2024-09-17T23:14:19.773 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:14:19.773 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:14:19.773 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:14:20.772 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-17T23:14:20.773 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:14:20.773 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:14:20.773 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:14:20.773 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=started 2024-09-17T23:14:20.773 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_result 2024-09-17T23:14:20.773 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=stopped 2024-09-17T23:14:20.774 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-17T23:14:20.774 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_instance=cluster1 2024-09-17T23:14:20.774 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:14:20.774 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ grep -i state 2024-09-17T23:14:20.774 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:14:20.774 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance=cluster1 2024-09-17T23:14:20.774 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local loop_instance 2024-09-17T23:14:20.774 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:14:20.775 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sleep 0 2024-09-17T23:14:20.775 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-17T23:14:20.777 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ seq 0 3 2024-09-17T23:14:20.777 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:14:20.777 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-17T23:14:20.778 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:0 2024-09-17T23:14:20.778 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:14:20.778 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:14:20.778 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-17T23:14:20.778 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:0 2024-09-17T23:14:20.778 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:14:20.778 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:14:20.778 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:14:20.778 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=0 2024-09-17T23:14:20.778 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-17T23:14:20.778 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:14:20.778 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:14:20.778 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=0 2024-09-17T23:14:20.778 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:14:20.779 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-17T23:14:20.779 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:0 2024-09-17T23:14:20.779 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:14:20.779 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:14:20.779 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-17T23:14:20.779 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:0 2024-09-17T23:14:20.779 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:14:20.779 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:14:20.779 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:14:20.779 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:14:20.779 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-17T23:14:20.779 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:14:20.779 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:14:20.779 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=0 2024-09-17T23:14:20.780 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=0 2024-09-17T23:14:20.780 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-17T23:14:20.799 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:14:20.800 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:14:20.800 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:14:20.800 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-17T23:14:20.882 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-17T23:14:20.882 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd mirror status : get status for rbd mirror 2024-09-17T23:14:20.882 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:admin_socket: invalid command 2024-09-17T23:14:20.885 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:14:20.885 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-17T23:14:20.885 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:1 2024-09-17T23:14:20.885 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:14:20.885 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:14:20.885 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-17T23:14:20.886 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:1 2024-09-17T23:14:20.886 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:14:20.886 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:14:20.886 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:14:20.886 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=1 2024-09-17T23:14:20.886 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-17T23:14:20.886 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:14:20.886 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:14:20.886 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=1 2024-09-17T23:14:20.886 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=1 2024-09-17T23:14:20.886 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-17T23:14:20.886 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:1 2024-09-17T23:14:20.886 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:14:20.886 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:14:20.887 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-17T23:14:20.887 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:1 2024-09-17T23:14:20.887 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:14:20.887 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:14:20.887 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:14:20.887 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=1 2024-09-17T23:14:20.887 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-17T23:14:20.887 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:14:20.887 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:14:20.887 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=1 2024-09-17T23:14:20.887 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=1 2024-09-17T23:14:20.887 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-17T23:14:20.905 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-17T23:14:20.905 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-17T23:14:20.905 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-17T23:14:20.905 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-17T23:14:20.981 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-17T23:14:20.982 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd mirror status : get status for rbd mirror 2024-09-17T23:14:20.982 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:admin_socket: invalid command 2024-09-17T23:14:20.984 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:14:20.984 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-17T23:14:20.985 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:2 2024-09-17T23:14:20.985 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:14:20.985 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:14:20.985 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-17T23:14:20.985 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:2 2024-09-17T23:14:20.985 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:14:20.985 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:14:20.985 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:14:20.985 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=2 2024-09-17T23:14:20.985 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-17T23:14:20.985 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:14:20.985 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:14:20.985 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=2 2024-09-17T23:14:20.986 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=2 2024-09-17T23:14:20.986 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-17T23:14:20.986 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:2 2024-09-17T23:14:20.986 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:14:20.986 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:14:20.986 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-17T23:14:20.986 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:2 2024-09-17T23:14:20.986 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:14:20.986 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:14:20.986 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:14:20.986 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=2 2024-09-17T23:14:20.986 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-17T23:14:20.986 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:14:20.986 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:14:20.987 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=2 2024-09-17T23:14:20.987 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=2 2024-09-17T23:14:20.987 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-17T23:14:21.004 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-17T23:14:21.004 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-17T23:14:21.004 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-17T23:14:21.004 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-17T23:14:21.079 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-17T23:14:21.079 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd mirror status : get status for rbd mirror 2024-09-17T23:14:21.079 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:admin_socket: invalid command 2024-09-17T23:14:21.082 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:14:21.082 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-09-17T23:14:21.082 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:3 2024-09-17T23:14:21.082 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:14:21.082 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:14:21.082 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-17T23:14:21.082 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:3 2024-09-17T23:14:21.083 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:14:21.083 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:14:21.083 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:14:21.083 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=3 2024-09-17T23:14:21.083 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-17T23:14:21.083 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:14:21.083 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:14:21.083 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=3 2024-09-17T23:14:21.083 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=3 2024-09-17T23:14:21.083 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-17T23:14:21.083 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:3 2024-09-17T23:14:21.083 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:14:21.083 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:14:21.083 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-17T23:14:21.084 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:3 2024-09-17T23:14:21.084 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:14:21.084 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:14:21.084 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:14:21.084 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=3 2024-09-17T23:14:21.084 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-17T23:14:21.084 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:14:21.084 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:14:21.084 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=3 2024-09-17T23:14:21.084 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=3 2024-09-17T23:14:21.084 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-17T23:14:21.101 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-17T23:14:21.101 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-17T23:14:21.101 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-17T23:14:21.101 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-09-17T23:14:21.185 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ return 0 2024-09-17T23:14:21.186 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ status_result=' "state": "Replaying"' 2024-09-17T23:14:21.186 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ echo ' "state": "Replaying"' 2024-09-17T23:14:21.186 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep -i Replaying 2024-09-17T23:14:21.187 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: "state": "Replaying" 2024-09-17T23:14:21.188 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_state=started 2024-09-17T23:14:21.188 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test started = started 2024-09-17T23:14:21.188 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:14:21.188 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-17T23:14:21.188 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:14:21.188 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:14:21.189 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:14:21.189 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:14:21.189 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = journal ']' 2024-09-17T23:14:21.189 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = snapshot ']' 2024-09-17T23:14:21.189 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-09-17T23:14:21.189 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:14:21.189 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:14:21.189 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:14:21.189 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:14:21.189 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster2-mirror-test.status 2024-09-17T23:14:21.189 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster2-mirror-test.status 2024-09-17T23:14:21.189 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:14:21.190 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-test.status 2024-09-17T23:14:21.190 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster1-mirror-test.status 2024-09-17T23:14:21.190 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster1-mirror-test.status 2024-09-17T23:14:21.190 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:14:21.191 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_status_log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-test.status 2024-09-17T23:14:21.191 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-17T23:14:21.191 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:14:21.191 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:14:21.191 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:14:21.191 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-17T23:14:22.786 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Snapshot ID: 29 2024-09-17T23:14:22.807 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.status 2024-09-17T23:14:22.810 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:14:22.811 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:14:22.811 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:14:22.811 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-test.status 2024-09-17T23:14:22.811 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-17T23:14:22.812 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:14:22.891 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-17T23:14:22.893 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snapshot_id=29 2024-09-17T23:14:22.893 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:14:22.893 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:14:22.893 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.2 2024-09-17T23:14:23.095 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.status 2024-09-17T23:14:23.095 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:14:23.095 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:14:23.095 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:14:23.095 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-test.status 2024-09-17T23:14:23.095 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-17T23:14:23.095 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:14:23.146 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:14:23.146 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:14:23.148 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:14:23.148 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:14:23.148 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:14:23.148 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:14:23.148 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 29 = '' 2024-09-17T23:14:23.148 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:14:23.148 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.4 2024-09-17T23:14:23.549 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.status 2024-09-17T23:14:23.550 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:14:23.550 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:14:23.550 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:14:23.550 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-test.status 2024-09-17T23:14:23.550 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-17T23:14:23.550 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:14:23.604 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:14:23.604 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:14:23.605 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:14:23.605 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:14:23.605 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:14:23.606 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:14:23.606 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 29 = '' 2024-09-17T23:14:23.606 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:14:23.606 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.8 2024-09-17T23:14:24.407 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.status 2024-09-17T23:14:24.407 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:14:24.407 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:14:24.408 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:14:24.408 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-test.status 2024-09-17T23:14:24.408 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-17T23:14:24.408 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:14:24.511 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:14:24.511 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:14:24.512 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:14:24.512 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:14:24.513 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:14:24.513 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:14:24.513 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 29 = '' 2024-09-17T23:14:24.513 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:14:24.513 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1.6 2024-09-17T23:14:26.115 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.status 2024-09-17T23:14:26.115 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:14:26.115 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:14:26.116 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:14:26.116 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-test.status 2024-09-17T23:14:26.116 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-17T23:14:26.116 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:14:26.172 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:14:26.175 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id=29 2024-09-17T23:14:26.175 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 29 = 29 2024-09-17T23:14:26.175 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:14:26.175 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snap_present cluster1 mirror test snap9 2024-09-17T23:14:26.175 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:14:26.176 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:14:26.176 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:14:26.176 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap9 2024-09-17T23:14:26.176 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:14:26.176 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-17T23:14:26.176 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:14:27.175 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 info mirror/test@snap9 2024-09-17T23:14:27.230 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'test': 2024-09-17T23:14:27.230 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 512 MiB in 128 objects 2024-09-17T23:14:27.231 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:14:27.231 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 18 2024-09-17T23:14:27.231 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 10c7e03f2edd 2024-09-17T23:14:27.231 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.10c7e03f2edd 2024-09-17T23:14:27.231 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:14:27.231 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-17T23:14:27.231 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:14:27.231 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:14:27.231 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:06:57 2024 2024-09-17T23:14:27.231 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:13:41 2024 2024-09-17T23:14:27.231 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:06:57 2024 2024-09-17T23:14:27.231 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: protected: False 2024-09-17T23:14:27.231 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:14:27.231 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:14:27.231 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: d6dfe272-2e04-4592-a96a-96648936ad18 2024-09-17T23:14:27.232 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: false 2024-09-17T23:14:27.234 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:14:27.234 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n snap8-clean ']' 2024-09-17T23:14:27.234 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ compare_image_snaps mirror test snap8-clean 2024-09-17T23:14:27.235 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:14:27.235 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:14:27.235 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap8-clean 2024-09-17T23:14:27.235 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local ret=0 2024-09-17T23:14:27.235 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local rmt_export=/tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export 2024-09-17T23:14:27.235 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local loc_export=/tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:14:27.235 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:14:27.235 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8-clean /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export 2024-09-17T23:14:27.758 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-17T23:14:27.767 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8-clean /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:14:28.462 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-17T23:14:28.470 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ cmp /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:14:28.717 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:14:28.836 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:14:28.836 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ compare_image_snaps mirror test snap9 2024-09-17T23:14:28.836 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:14:28.836 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:14:28.836 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap9 2024-09-17T23:14:28.836 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local ret=0 2024-09-17T23:14:28.836 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local rmt_export=/tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export 2024-09-17T23:14:28.837 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local loc_export=/tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:14:28.837 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:14:28.837 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9 /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export 2024-09-17T23:14:29.369 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-17T23:14:29.380 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9 /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:14:30.089 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-17T23:14:30.098 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ cmp /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:14:30.346 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:14:30.467 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:14:30.467 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ clean_snap_name=snap9-clean 2024-09-17T23:14:30.467 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_snap cluster2 mirror test snap9-clean 2024-09-17T23:14:30.467 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:14:30.467 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:14:30.467 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:14:30.468 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap9-clean 2024-09-17T23:14:30.468 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap9-clean --debug-rbd=20 --debug-journaler=20 2024-09-17T23:14:30.808 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 10` 2024-09-17T23:14:30.808 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ stress_write_image cluster2 mirror test 2024-09-17T23:14:30.808 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:14:30.808 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:14:30.808 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:14:30.808 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-17T23:14:30.809 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local duration=8 2024-09-17T23:14:30.810 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set +e 2024-09-17T23:14:30.810 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-17T23:14:31.131 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:100 IOs, 50867200 bytes 2024-09-17T23:14:31.301 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:200 IOs, 83853312 bytes 2024-09-17T23:14:31.437 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:300 IOs, 130980864 bytes 2024-09-17T23:14:31.525 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:400 IOs, 137134080 bytes 2024-09-17T23:14:31.623 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:500 IOs, 164272128 bytes 2024-09-17T23:14:31.757 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:600 IOs, 233400320 bytes 2024-09-17T23:14:31.837 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:700 IOs, 240538624 bytes 2024-09-17T23:14:31.911 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:800 IOs, 243508224 bytes 2024-09-17T23:14:32.018 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:900 IOs, 283611136 bytes 2024-09-17T23:14:32.116 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1000 IOs, 302251008 bytes 2024-09-17T23:14:32.243 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1100 IOs, 392337408 bytes 2024-09-17T23:14:32.388 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1200 IOs, 445484032 bytes 2024-09-17T23:14:32.492 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1300 IOs, 494643200 bytes 2024-09-17T23:14:32.607 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1400 IOs, 536751104 bytes 2024-09-17T23:14:32.751 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1500 IOs, 603353088 bytes 2024-09-17T23:14:32.846 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1600 IOs, 627690496 bytes 2024-09-17T23:14:32.973 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1700 IOs, 709807104 bytes 2024-09-17T23:14:33.109 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1800 IOs, 750541824 bytes 2024-09-17T23:14:33.182 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1900 IOs, 765921280 bytes 2024-09-17T23:14:33.328 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2000 IOs, 839856128 bytes 2024-09-17T23:14:33.423 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2100 IOs, 885005312 bytes 2024-09-17T23:14:33.578 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2200 IOs, 959899648 bytes 2024-09-17T23:14:33.670 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2300 IOs, 982159360 bytes 2024-09-17T23:14:33.799 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2400 IOs, 1040968704 bytes 2024-09-17T23:14:33.921 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2500 IOs, 1104333824 bytes 2024-09-17T23:14:34.005 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2600 IOs, 1110835200 bytes 2024-09-17T23:14:34.126 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2700 IOs, 1169828864 bytes 2024-09-17T23:14:34.278 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2800 IOs, 1255432192 bytes 2024-09-17T23:14:34.393 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2900 IOs, 1302191104 bytes 2024-09-17T23:14:34.551 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3000 IOs, 1400080384 bytes 2024-09-17T23:14:34.652 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3100 IOs, 1431729152 bytes 2024-09-17T23:14:34.811 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3200 IOs, 1533319168 bytes 2024-09-17T23:14:34.933 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3300 IOs, 1589514240 bytes 2024-09-17T23:14:35.009 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3400 IOs, 1590100992 bytes 2024-09-17T23:14:35.121 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3500 IOs, 1631731712 bytes 2024-09-17T23:14:35.300 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3600 IOs, 1749180416 bytes 2024-09-17T23:14:35.412 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3700 IOs, 1794641920 bytes 2024-09-17T23:14:35.499 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3800 IOs, 1810503680 bytes 2024-09-17T23:14:35.622 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3900 IOs, 1868924928 bytes 2024-09-17T23:14:35.729 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4000 IOs, 1912345600 bytes 2024-09-17T23:14:35.847 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4100 IOs, 1963476992 bytes 2024-09-17T23:14:35.965 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4200 IOs, 2006995968 bytes 2024-09-17T23:14:36.038 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4300 IOs, 2010458112 bytes 2024-09-17T23:14:36.166 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4400 IOs, 2071170048 bytes 2024-09-17T23:14:36.278 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4500 IOs, 2126080000 bytes 2024-09-17T23:14:36.381 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4600 IOs, 2155320320 bytes 2024-09-17T23:14:36.492 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4700 IOs, 2208218112 bytes 2024-09-17T23:14:36.571 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4800 IOs, 2208838656 bytes 2024-09-17T23:14:36.683 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4900 IOs, 2255228928 bytes 2024-09-17T23:14:36.764 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5000 IOs, 2266514432 bytes 2024-09-17T23:14:36.853 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5100 IOs, 2292391936 bytes 2024-09-17T23:14:36.951 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5200 IOs, 2313090048 bytes 2024-09-17T23:14:37.078 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5300 IOs, 2368014336 bytes 2024-09-17T23:14:37.223 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5400 IOs, 2441042944 bytes 2024-09-17T23:14:37.333 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5500 IOs, 2478950400 bytes 2024-09-17T23:14:37.478 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5600 IOs, 2557334528 bytes 2024-09-17T23:14:37.615 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5700 IOs, 2625198080 bytes 2024-09-17T23:14:37.760 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5800 IOs, 2703548416 bytes 2024-09-17T23:14:37.854 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5900 IOs, 2732526592 bytes 2024-09-17T23:14:37.946 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6000 IOs, 2752082944 bytes 2024-09-17T23:14:38.038 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6100 IOs, 2773707776 bytes 2024-09-17T23:14:38.127 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6200 IOs, 2794618880 bytes 2024-09-17T23:14:38.216 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6300 IOs, 2817958912 bytes 2024-09-17T23:14:38.297 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6400 IOs, 2823794688 bytes 2024-09-17T23:14:38.444 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6500 IOs, 2916305920 bytes 2024-09-17T23:14:38.558 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6600 IOs, 2947378176 bytes 2024-09-17T23:14:38.652 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6700 IOs, 2970136576 bytes 2024-09-17T23:14:38.773 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6800 IOs, 3029052416 bytes 2024-09-17T23:14:38.818 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ error_code=124 2024-09-17T23:14:38.818 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set -e 2024-09-17T23:14:38.818 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' 124 -eq 124 ']' 2024-09-17T23:14:38.819 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:14:38.819 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-17T23:14:38.819 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:14:38.819 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:14:38.819 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:14:38.819 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state_pattern=up+replaying 2024-09-17T23:14:38.819 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local description_pattern= 2024-09-17T23:14:38.819 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local service_pattern= 2024-09-17T23:14:38.819 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:14:38.819 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:14:39.820 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-17T23:14:39.820 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:14:39.820 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:14:39.820 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:14:39.820 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state_pattern=up+replaying 2024-09-17T23:14:39.820 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local description_pattern= 2024-09-17T23:14:39.820 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local service_pattern= 2024-09-17T23:14:39.820 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status 2024-09-17T23:14:39.820 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ CEPH_ARGS= 2024-09-17T23:14:39.821 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-17T23:14:39.881 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ status='test: 2024-09-17T23:14:39.881 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: global_id: d6dfe272-2e04-4592-a96a-96648936ad18 2024-09-17T23:14:39.881 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: state: up+replaying 2024-09-17T23:14:39.881 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: description: replaying, {"bytes_per_second":16898833.07,"bytes_per_snapshot":522376704.0,"last_snapshot_bytes":519016448,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726614861,"remote_snapshot_timestamp":1726614861,"replay_state":"idle"} 2024-09-17T23:14:39.881 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: service: mirror.3 on smithi038 2024-09-17T23:14:39.881 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: last_update: 2024-09-17 23:14:25 2024-09-17T23:14:39.881 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: peer_sites: 2024-09-17T23:14:39.881 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: name: cluster2 2024-09-17T23:14:39.881 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: state: up+stopped 2024-09-17T23:14:39.881 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: description: local image is primary 2024-09-17T23:14:39.882 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: last_update: 2024-09-17 23:14:26' 2024-09-17T23:14:39.882 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep '^ state: .*up+replaying' 2024-09-17T23:14:39.882 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: state: up+replaying 2024-09-17T23:14:39.882 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep '^ description: .*' 2024-09-17T23:14:39.883 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: description: replaying, {"bytes_per_second":16898833.07,"bytes_per_snapshot":522376704.0,"last_snapshot_bytes":519016448,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726614861,"remote_snapshot_timestamp":1726614861,"replay_state":"idle"} 2024-09-17T23:14:39.884 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n '' ']' 2024-09-17T23:14:39.884 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ echo up+replaying 2024-09-17T23:14:39.884 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep '^up+' 2024-09-17T23:14:39.884 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:up+replaying 2024-09-17T23:14:39.885 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:14:39.886 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: service: mirror.3 on smithi038 2024-09-17T23:14:39.886 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local last_update 2024-09-17T23:14:39.886 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:14:39.888 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ last_update='2024-09-17 23:14:25' 2024-09-17T23:14:39.889 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-17 23:14:25' 2024-09-17T23:14:39.889 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:14:39.889 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:14:39.889 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:14:39.889 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state_pattern=up+replaying 2024-09-17T23:14:39.889 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local 'prev_last_update=2024-09-17 23:14:25' 2024-09-17T23:14:39.889 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status 2024-09-17T23:14:39.889 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ CEPH_ARGS= 2024-09-17T23:14:39.889 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-17T23:14:39.951 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ status='OKOKOK142214298mirror.2smithi03819.1.1-330-gc2ddf972falseOK42234294mirror.3smithi03819.1.1-330-gc2ddf972trueOK42624310mirror.1smithi03819.1.1-330-gc2ddf972falseOK45584561mirror.0smithi03819.1.1-330-gc2ddf972falseOKtestd6dfe272-2e04-4592-a96a-96648936ad18up+replayingreplaying, {"bytes_per_second":16898833.07,"bytes_per_snapshot":522376704.0,"last_snapshot_bytes":519016448,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726614861,"remote_snapshot_timestamp":1726614861,"replay_state":"idle"}42234294mirror.3smithi0382024-09-17 23:14:25cluster20ba5654d-47fb-4672-b20c-7359a1340314up+stoppedlocal image is primary2024-09-17 23:14:26' 2024-09-17T23:14:39.951 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local last_update state 2024-09-17T23:14:39.951 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-17T23:14:39.952 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ last_update='2024-09-17 23:14:25' 2024-09-17T23:14:39.953 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-17T23:14:39.955 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ state=up+replaying 2024-09-17T23:14:39.955 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ echo up+replaying 2024-09-17T23:14:39.955 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep up+replaying 2024-09-17T23:14:39.956 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:up+replaying 2024-09-17T23:14:39.957 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:14:39.957 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:14:39.957 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ snap_name=snap10 2024-09-17T23:14:39.957 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_snap cluster2 mirror test snap10 2024-09-17T23:14:39.957 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:14:39.957 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:14:39.957 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:14:39.957 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap10 2024-09-17T23:14:39.957 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap10 --debug-rbd=20 --debug-journaler=20 2024-09-17T23:14:54.806 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-09-17T23:14:54.807 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-09-17T23:14:54.807 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2024-09-17T23:14:54.808 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi038.stderr:2024-09-17T23:14:54.805+0000 7f3f64ba9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 42201) UID: 1000 2024-09-17T23:15:00.808 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-09-17T23:15:00.808 INFO:teuthology.orchestra.run:waiting for 600 2024-09-17T23:15:00.809 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-09-17T23:15:00.809 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-09-17T23:15:00.809 DEBUG:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-09-17T23:15:00.812 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-09-17T23:15:00.813 INFO:tasks.rbd_mirror_thrash:waiting for 80 secs before thrashing 2024-09-17T23:15:07.105 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-17T23:15:07.105 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:15:07.105 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:15:07.105 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:15:07.105 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-17T23:15:07.105 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:15:07.106 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:15:07.106 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:15:07.106 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=started 2024-09-17T23:15:07.106 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:15:07.106 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:15:07.106 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:15:08.106 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-17T23:15:08.106 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:15:08.107 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:15:08.107 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:15:08.107 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=started 2024-09-17T23:15:08.107 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_result 2024-09-17T23:15:08.107 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=stopped 2024-09-17T23:15:08.111 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-17T23:15:08.112 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_instance=cluster1 2024-09-17T23:15:08.112 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ grep -i state 2024-09-17T23:15:08.112 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:15:08.112 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:15:08.112 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance=cluster1 2024-09-17T23:15:08.112 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local loop_instance 2024-09-17T23:15:08.112 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:15:08.112 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sleep 0 2024-09-17T23:15:08.112 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-17T23:15:08.112 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ seq 0 3 2024-09-17T23:15:08.112 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:15:08.112 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-17T23:15:08.112 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:0 2024-09-17T23:15:08.113 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:15:08.113 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:15:08.113 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-17T23:15:08.113 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:0 2024-09-17T23:15:08.113 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:15:08.113 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:15:08.113 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:15:08.113 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=0 2024-09-17T23:15:08.113 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-17T23:15:08.113 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:15:08.113 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:15:08.113 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=0 2024-09-17T23:15:08.114 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:15:08.114 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-17T23:15:08.114 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:0 2024-09-17T23:15:08.114 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:15:08.114 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:15:08.114 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-17T23:15:08.115 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:0 2024-09-17T23:15:08.115 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:15:08.116 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:15:08.116 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:15:08.116 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:15:08.116 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-17T23:15:08.116 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:15:08.116 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:15:08.116 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=0 2024-09-17T23:15:08.116 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=0 2024-09-17T23:15:08.116 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-17T23:15:08.134 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:15:08.134 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:15:08.134 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:15:08.134 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-17T23:15:08.215 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-17T23:15:08.215 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd mirror status : get status for rbd mirror 2024-09-17T23:15:08.216 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:admin_socket: invalid command 2024-09-17T23:15:08.219 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:15:08.219 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-17T23:15:08.219 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:1 2024-09-17T23:15:08.219 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:15:08.219 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:15:08.219 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-17T23:15:08.219 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:1 2024-09-17T23:15:08.219 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:15:08.219 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:15:08.219 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:15:08.219 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=1 2024-09-17T23:15:08.220 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-17T23:15:08.220 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:15:08.220 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:15:08.220 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=1 2024-09-17T23:15:08.220 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=1 2024-09-17T23:15:08.220 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-17T23:15:08.220 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:1 2024-09-17T23:15:08.220 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:15:08.220 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:15:08.220 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-17T23:15:08.220 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:1 2024-09-17T23:15:08.220 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:15:08.220 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:15:08.220 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:15:08.221 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=1 2024-09-17T23:15:08.221 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-17T23:15:08.221 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:15:08.221 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:15:08.221 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=1 2024-09-17T23:15:08.221 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=1 2024-09-17T23:15:08.221 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-17T23:15:08.239 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-17T23:15:08.239 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-17T23:15:08.239 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-17T23:15:08.239 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-17T23:15:08.315 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-17T23:15:08.315 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd mirror status : get status for rbd mirror 2024-09-17T23:15:08.315 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:admin_socket: invalid command 2024-09-17T23:15:08.318 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:15:08.318 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-17T23:15:08.318 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:2 2024-09-17T23:15:08.318 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:15:08.318 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:15:08.318 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-17T23:15:08.318 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:2 2024-09-17T23:15:08.319 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:15:08.319 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:15:08.319 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:15:08.319 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=2 2024-09-17T23:15:08.319 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-17T23:15:08.319 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:15:08.319 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:15:08.319 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=2 2024-09-17T23:15:08.319 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=2 2024-09-17T23:15:08.319 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-17T23:15:08.319 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:2 2024-09-17T23:15:08.319 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:15:08.319 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:15:08.320 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-17T23:15:08.320 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:2 2024-09-17T23:15:08.320 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:15:08.320 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:15:08.320 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:15:08.320 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=2 2024-09-17T23:15:08.320 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-17T23:15:08.320 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:15:08.320 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:15:08.320 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=2 2024-09-17T23:15:08.320 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=2 2024-09-17T23:15:08.320 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-17T23:15:08.337 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-17T23:15:08.337 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-17T23:15:08.337 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-17T23:15:08.337 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-17T23:15:08.412 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-17T23:15:08.412 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd mirror status : get status for rbd mirror 2024-09-17T23:15:08.413 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:admin_socket: invalid command 2024-09-17T23:15:08.415 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:15:08.415 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-09-17T23:15:08.415 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:3 2024-09-17T23:15:08.415 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:15:08.415 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:15:08.415 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-17T23:15:08.415 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:3 2024-09-17T23:15:08.416 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:15:08.416 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:15:08.416 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:15:08.416 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=3 2024-09-17T23:15:08.416 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-17T23:15:08.416 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:15:08.416 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:15:08.416 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=3 2024-09-17T23:15:08.416 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=3 2024-09-17T23:15:08.416 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-17T23:15:08.416 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:3 2024-09-17T23:15:08.416 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:15:08.416 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:15:08.416 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-17T23:15:08.417 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:3 2024-09-17T23:15:08.417 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:15:08.417 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:15:08.417 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:15:08.417 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=3 2024-09-17T23:15:08.417 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-17T23:15:08.417 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:15:08.417 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:15:08.417 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=3 2024-09-17T23:15:08.417 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=3 2024-09-17T23:15:08.417 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-17T23:15:08.435 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-17T23:15:08.435 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-17T23:15:08.435 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-17T23:15:08.435 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-09-17T23:15:08.519 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ return 0 2024-09-17T23:15:08.519 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ status_result=' "state": "Replaying"' 2024-09-17T23:15:08.519 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ echo ' "state": "Replaying"' 2024-09-17T23:15:08.520 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep -i Replaying 2024-09-17T23:15:08.520 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: "state": "Replaying" 2024-09-17T23:15:08.522 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_state=started 2024-09-17T23:15:08.522 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test started = started 2024-09-17T23:15:08.522 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:15:08.522 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-17T23:15:08.522 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:15:08.522 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:15:08.522 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:15:08.522 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:15:08.523 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = journal ']' 2024-09-17T23:15:08.523 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = snapshot ']' 2024-09-17T23:15:08.523 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-09-17T23:15:08.523 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:15:08.523 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:15:08.523 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:15:08.523 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:15:08.523 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster2-mirror-test.status 2024-09-17T23:15:08.523 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster2-mirror-test.status 2024-09-17T23:15:08.523 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:15:08.524 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-test.status 2024-09-17T23:15:08.524 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster1-mirror-test.status 2024-09-17T23:15:08.525 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster1-mirror-test.status 2024-09-17T23:15:08.525 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:15:08.525 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_status_log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-test.status 2024-09-17T23:15:08.525 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-17T23:15:08.525 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:15:08.525 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:15:08.525 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:15:08.525 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-17T23:15:10.092 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Snapshot ID: 32 2024-09-17T23:15:10.118 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.status 2024-09-17T23:15:10.118 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:15:10.119 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:15:10.119 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:15:10.119 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-test.status 2024-09-17T23:15:10.119 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-17T23:15:10.119 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:15:10.203 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-17T23:15:10.205 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snapshot_id=32 2024-09-17T23:15:10.205 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:15:10.205 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:15:10.205 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.2 2024-09-17T23:15:10.407 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.status 2024-09-17T23:15:10.407 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:15:10.407 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:15:10.407 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:15:10.407 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-test.status 2024-09-17T23:15:10.407 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-17T23:15:10.407 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:15:10.460 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:15:10.460 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:15:10.461 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:15:10.461 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:15:10.461 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:15:10.462 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:15:10.462 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 32 = '' 2024-09-17T23:15:10.462 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:15:10.462 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.4 2024-09-17T23:15:10.864 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.status 2024-09-17T23:15:10.864 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:15:10.864 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:15:10.864 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:15:10.864 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-test.status 2024-09-17T23:15:10.864 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-17T23:15:10.864 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:15:10.920 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:15:10.920 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:15:10.921 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:15:10.922 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:15:10.922 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:15:10.922 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:15:10.922 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 32 = '' 2024-09-17T23:15:10.922 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:15:10.923 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.8 2024-09-17T23:15:11.724 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.status 2024-09-17T23:15:11.724 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:15:11.724 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:15:11.724 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:15:11.724 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-test.status 2024-09-17T23:15:11.724 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-17T23:15:11.724 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:15:11.814 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:15:11.815 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:15:11.817 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:15:11.817 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:15:11.817 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:15:11.817 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:15:11.818 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 32 = '' 2024-09-17T23:15:11.818 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:15:11.818 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1.6 2024-09-17T23:15:13.419 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.status 2024-09-17T23:15:13.419 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:15:13.419 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:15:13.419 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:15:13.419 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-test.status 2024-09-17T23:15:13.420 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-17T23:15:13.420 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:15:13.477 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:15:13.479 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id=32 2024-09-17T23:15:13.479 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 32 = 32 2024-09-17T23:15:13.479 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:15:13.479 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snap_present cluster1 mirror test snap10 2024-09-17T23:15:13.479 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:15:13.479 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:15:13.480 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:15:13.480 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap10 2024-09-17T23:15:13.480 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:15:13.480 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-17T23:15:13.480 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:15:14.480 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 info mirror/test@snap10 2024-09-17T23:15:14.532 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'test': 2024-09-17T23:15:14.532 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 512 MiB in 128 objects 2024-09-17T23:15:14.532 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:15:14.532 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 20 2024-09-17T23:15:14.532 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 10c7e03f2edd 2024-09-17T23:15:14.533 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.10c7e03f2edd 2024-09-17T23:15:14.533 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:15:14.533 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-17T23:15:14.533 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:15:14.533 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:15:14.533 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:06:57 2024 2024-09-17T23:15:14.533 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:13:41 2024 2024-09-17T23:15:14.533 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:06:57 2024 2024-09-17T23:15:14.533 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: protected: False 2024-09-17T23:15:14.533 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:15:14.533 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:15:14.533 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: d6dfe272-2e04-4592-a96a-96648936ad18 2024-09-17T23:15:14.533 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: false 2024-09-17T23:15:14.536 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:15:14.536 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n snap9-clean ']' 2024-09-17T23:15:14.536 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ compare_image_snaps mirror test snap9-clean 2024-09-17T23:15:14.536 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:15:14.536 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:15:14.537 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap9-clean 2024-09-17T23:15:14.537 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local ret=0 2024-09-17T23:15:14.537 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local rmt_export=/tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export 2024-09-17T23:15:14.537 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local loc_export=/tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:15:14.537 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:15:14.537 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9-clean /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export 2024-09-17T23:15:15.063 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-17T23:15:15.072 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9-clean /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:15:15.775 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-17T23:15:15.782 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ cmp /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:15:16.029 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:15:16.148 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:15:16.148 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ compare_image_snaps mirror test snap10 2024-09-17T23:15:16.148 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:15:16.148 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:15:16.148 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap10 2024-09-17T23:15:16.148 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local ret=0 2024-09-17T23:15:16.149 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local rmt_export=/tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export 2024-09-17T23:15:16.149 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local loc_export=/tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:15:16.149 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:15:16.149 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10 /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export 2024-09-17T23:15:16.665 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-17T23:15:16.676 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10 /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:15:17.398 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-17T23:15:17.407 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ cmp /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:15:17.654 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:15:17.773 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:15:17.773 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ clean_snap_name=snap10-clean 2024-09-17T23:15:17.773 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_snap cluster2 mirror test snap10-clean 2024-09-17T23:15:17.773 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:15:17.773 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:15:17.773 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:15:17.774 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap10-clean 2024-09-17T23:15:17.774 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap10-clean --debug-rbd=20 --debug-journaler=20 2024-09-17T23:15:18.124 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-17T23:15:18.124 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:15:18.124 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:15:18.125 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:15:18.125 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-17T23:15:18.125 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:15:18.125 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:15:18.125 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:15:18.125 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=started 2024-09-17T23:15:18.125 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:15:18.125 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:15:18.125 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:15:19.125 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-17T23:15:19.125 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:15:19.126 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:15:19.126 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:15:19.126 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=started 2024-09-17T23:15:19.126 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_result 2024-09-17T23:15:19.126 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=stopped 2024-09-17T23:15:19.126 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-17T23:15:19.127 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_instance=cluster1 2024-09-17T23:15:19.127 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:15:19.127 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ grep -i state 2024-09-17T23:15:19.127 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:15:19.127 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance=cluster1 2024-09-17T23:15:19.127 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local loop_instance 2024-09-17T23:15:19.127 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:15:19.127 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sleep 0 2024-09-17T23:15:19.130 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-17T23:15:19.130 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ seq 0 3 2024-09-17T23:15:19.130 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:15:19.130 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-17T23:15:19.130 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:0 2024-09-17T23:15:19.130 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:15:19.130 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:15:19.130 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-17T23:15:19.130 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:0 2024-09-17T23:15:19.130 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:15:19.130 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:15:19.131 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:15:19.131 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=0 2024-09-17T23:15:19.131 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-17T23:15:19.131 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:15:19.131 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:15:19.131 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=0 2024-09-17T23:15:19.131 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:15:19.131 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-17T23:15:19.131 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:0 2024-09-17T23:15:19.131 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:15:19.131 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:15:19.131 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-17T23:15:19.131 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:0 2024-09-17T23:15:19.131 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:15:19.131 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:15:19.132 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:15:19.132 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:15:19.132 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-17T23:15:19.132 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:15:19.132 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:15:19.132 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=0 2024-09-17T23:15:19.133 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=0 2024-09-17T23:15:19.133 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-17T23:15:19.152 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:15:19.152 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:15:19.152 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:15:19.152 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-17T23:15:19.233 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-17T23:15:19.233 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd mirror status : get status for rbd mirror 2024-09-17T23:15:19.233 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:admin_socket: invalid command 2024-09-17T23:15:19.236 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:15:19.236 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-17T23:15:19.236 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:1 2024-09-17T23:15:19.236 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:15:19.236 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:15:19.236 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-17T23:15:19.236 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:1 2024-09-17T23:15:19.236 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:15:19.236 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:15:19.236 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:15:19.237 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=1 2024-09-17T23:15:19.237 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-17T23:15:19.237 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:15:19.237 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:15:19.237 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=1 2024-09-17T23:15:19.237 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=1 2024-09-17T23:15:19.237 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-17T23:15:19.237 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:1 2024-09-17T23:15:19.237 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:15:19.238 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:15:19.238 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-17T23:15:19.238 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:1 2024-09-17T23:15:19.238 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:15:19.238 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:15:19.239 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:15:19.239 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=1 2024-09-17T23:15:19.239 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-17T23:15:19.239 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:15:19.239 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:15:19.239 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=1 2024-09-17T23:15:19.239 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=1 2024-09-17T23:15:19.240 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-17T23:15:19.255 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-17T23:15:19.255 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-17T23:15:19.255 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-17T23:15:19.256 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-17T23:15:19.331 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-17T23:15:19.331 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd mirror status : get status for rbd mirror 2024-09-17T23:15:19.331 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:admin_socket: invalid command 2024-09-17T23:15:19.334 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:15:19.334 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-17T23:15:19.335 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:2 2024-09-17T23:15:19.335 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:15:19.335 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:15:19.335 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-17T23:15:19.335 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:2 2024-09-17T23:15:19.335 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:15:19.335 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:15:19.335 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:15:19.335 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=2 2024-09-17T23:15:19.335 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-17T23:15:19.335 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:15:19.335 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:15:19.335 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=2 2024-09-17T23:15:19.335 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=2 2024-09-17T23:15:19.336 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-17T23:15:19.336 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:2 2024-09-17T23:15:19.336 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:15:19.336 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:15:19.336 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-17T23:15:19.336 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:2 2024-09-17T23:15:19.336 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:15:19.336 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:15:19.336 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:15:19.336 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=2 2024-09-17T23:15:19.336 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-17T23:15:19.336 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:15:19.337 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:15:19.337 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=2 2024-09-17T23:15:19.337 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=2 2024-09-17T23:15:19.337 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-17T23:15:19.353 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-17T23:15:19.353 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-17T23:15:19.353 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-17T23:15:19.353 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-17T23:15:19.428 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-17T23:15:19.429 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd mirror status : get status for rbd mirror 2024-09-17T23:15:19.429 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:admin_socket: invalid command 2024-09-17T23:15:19.432 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:15:19.432 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-09-17T23:15:19.432 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:3 2024-09-17T23:15:19.432 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:15:19.432 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:15:19.432 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-17T23:15:19.432 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:3 2024-09-17T23:15:19.432 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:15:19.432 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:15:19.432 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:15:19.432 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=3 2024-09-17T23:15:19.432 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-17T23:15:19.432 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:15:19.433 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:15:19.433 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=3 2024-09-17T23:15:19.433 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=3 2024-09-17T23:15:19.433 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-17T23:15:19.433 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:3 2024-09-17T23:15:19.433 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:15:19.433 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:15:19.433 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-17T23:15:19.433 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:3 2024-09-17T23:15:19.433 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:15:19.433 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:15:19.433 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:15:19.433 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=3 2024-09-17T23:15:19.433 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-17T23:15:19.433 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:15:19.434 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:15:19.434 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=3 2024-09-17T23:15:19.434 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=3 2024-09-17T23:15:19.434 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-17T23:15:19.450 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-17T23:15:19.451 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-17T23:15:19.451 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-17T23:15:19.451 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-09-17T23:15:19.535 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ return 0 2024-09-17T23:15:19.535 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ status_result=' "state": "Replaying"' 2024-09-17T23:15:19.535 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ echo ' "state": "Replaying"' 2024-09-17T23:15:19.535 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep -i Replaying 2024-09-17T23:15:19.536 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: "state": "Replaying" 2024-09-17T23:15:19.537 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_state=started 2024-09-17T23:15:19.537 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test started = started 2024-09-17T23:15:19.537 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:15:19.537 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-17T23:15:19.538 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:15:19.538 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:15:19.538 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:15:19.538 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:15:19.538 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = journal ']' 2024-09-17T23:15:19.538 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = snapshot ']' 2024-09-17T23:15:19.538 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-09-17T23:15:19.538 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:15:19.538 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:15:19.538 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:15:19.538 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:15:19.538 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster2-mirror-test.status 2024-09-17T23:15:19.538 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster2-mirror-test.status 2024-09-17T23:15:19.538 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:15:19.539 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-test.status 2024-09-17T23:15:19.539 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster1-mirror-test.status 2024-09-17T23:15:19.539 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster1-mirror-test.status 2024-09-17T23:15:19.539 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:15:19.540 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_status_log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-test.status 2024-09-17T23:15:19.540 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-17T23:15:19.541 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:15:19.541 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:15:19.541 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:15:19.541 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-17T23:15:21.127 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Snapshot ID: 34 2024-09-17T23:15:21.154 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.status 2024-09-17T23:15:21.154 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:15:21.155 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:15:21.155 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:15:21.155 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-test.status 2024-09-17T23:15:21.155 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:15:21.155 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-17T23:15:21.243 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-17T23:15:21.245 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snapshot_id=34 2024-09-17T23:15:21.245 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:15:21.246 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:15:21.246 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.2 2024-09-17T23:15:21.447 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.status 2024-09-17T23:15:21.447 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:15:21.447 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:15:21.447 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:15:21.447 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-test.status 2024-09-17T23:15:21.447 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-17T23:15:21.447 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:15:21.503 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:15:21.503 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:15:21.505 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:15:21.505 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:15:21.505 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:15:21.505 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:15:21.505 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 34 = '' 2024-09-17T23:15:21.505 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:15:21.506 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.4 2024-09-17T23:15:21.906 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.status 2024-09-17T23:15:21.907 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:15:21.907 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:15:21.907 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:15:21.907 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-test.status 2024-09-17T23:15:21.907 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-17T23:15:21.907 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:15:21.965 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:15:21.965 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:15:21.966 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:15:21.967 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:15:21.967 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:15:21.967 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:15:21.967 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 34 = '' 2024-09-17T23:15:21.968 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:15:21.968 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.8 2024-09-17T23:15:22.769 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.status 2024-09-17T23:15:22.769 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:15:22.769 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:15:22.769 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:15:22.769 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-test.status 2024-09-17T23:15:22.769 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-17T23:15:22.770 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:15:22.826 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:15:22.829 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id=34 2024-09-17T23:15:22.829 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 34 = 34 2024-09-17T23:15:22.829 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:15:22.829 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snap_present cluster1 mirror test snap10-clean 2024-09-17T23:15:22.829 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:15:22.829 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:15:22.829 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:15:22.829 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap10-clean 2024-09-17T23:15:22.830 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:15:22.830 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-17T23:15:22.830 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:15:23.830 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 info mirror/test@snap10-clean 2024-09-17T23:15:23.881 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'test': 2024-09-17T23:15:23.882 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 512 MiB in 128 objects 2024-09-17T23:15:23.882 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:15:23.882 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 21 2024-09-17T23:15:23.882 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 10c7e03f2edd 2024-09-17T23:15:23.882 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.10c7e03f2edd 2024-09-17T23:15:23.882 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:15:23.882 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-17T23:15:23.882 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:15:23.882 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:15:23.882 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:06:57 2024 2024-09-17T23:15:23.882 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:15:15 2024 2024-09-17T23:15:23.883 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:06:57 2024 2024-09-17T23:15:23.883 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: protected: False 2024-09-17T23:15:23.883 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:15:23.883 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:15:23.883 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: d6dfe272-2e04-4592-a96a-96648936ad18 2024-09-17T23:15:23.883 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: false 2024-09-17T23:15:23.886 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:15:23.886 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ seq 1 10 2024-09-17T23:15:23.887 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 10` 2024-09-17T23:15:23.887 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ snap_name=snap1 2024-09-17T23:15:23.887 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ compare_image_snaps mirror test snap1 2024-09-17T23:15:23.888 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:15:23.888 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:15:23.888 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap1 2024-09-17T23:15:23.888 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local ret=0 2024-09-17T23:15:23.888 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local rmt_export=/tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export 2024-09-17T23:15:23.888 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local loc_export=/tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:15:23.888 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:15:23.888 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1 /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export 2024-09-17T23:15:24.412 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-17T23:15:24.422 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1 /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:15:25.173 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-17T23:15:25.182 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ cmp /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:15:25.428 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:15:25.549 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:15:25.549 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ snap_name=snap1-clean 2024-09-17T23:15:25.549 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ compare_image_snaps mirror test snap1-clean 2024-09-17T23:15:25.549 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:15:25.550 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:15:25.550 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap1-clean 2024-09-17T23:15:25.550 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local ret=0 2024-09-17T23:15:25.550 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local rmt_export=/tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export 2024-09-17T23:15:25.550 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local loc_export=/tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:15:25.550 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:15:25.550 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1-clean /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export 2024-09-17T23:15:26.049 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-17T23:15:26.058 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1-clean /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:15:26.730 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-17T23:15:26.739 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ cmp /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:15:26.986 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:15:27.110 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:15:27.110 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 10` 2024-09-17T23:15:27.110 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ snap_name=snap2 2024-09-17T23:15:27.110 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ compare_image_snaps mirror test snap2 2024-09-17T23:15:27.111 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:15:27.111 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:15:27.111 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap2 2024-09-17T23:15:27.111 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local ret=0 2024-09-17T23:15:27.111 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local rmt_export=/tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export 2024-09-17T23:15:27.111 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local loc_export=/tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:15:27.111 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:15:27.111 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2 /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export 2024-09-17T23:15:27.628 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-17T23:15:27.639 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2 /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:15:28.339 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-17T23:15:28.347 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ cmp /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:15:28.593 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:15:28.713 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:15:28.713 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ snap_name=snap2-clean 2024-09-17T23:15:28.713 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ compare_image_snaps mirror test snap2-clean 2024-09-17T23:15:28.713 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:15:28.713 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:15:28.713 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap2-clean 2024-09-17T23:15:28.713 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local ret=0 2024-09-17T23:15:28.714 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local rmt_export=/tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export 2024-09-17T23:15:28.714 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local loc_export=/tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:15:28.714 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:15:28.714 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2-clean /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export 2024-09-17T23:15:29.239 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-17T23:15:29.250 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2-clean /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:15:29.959 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-17T23:15:29.967 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ cmp /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:15:30.216 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:15:30.336 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:15:30.337 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 10` 2024-09-17T23:15:30.337 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ snap_name=snap3 2024-09-17T23:15:30.337 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ compare_image_snaps mirror test snap3 2024-09-17T23:15:30.337 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:15:30.337 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:15:30.337 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap3 2024-09-17T23:15:30.337 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local ret=0 2024-09-17T23:15:30.337 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local rmt_export=/tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export 2024-09-17T23:15:30.338 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local loc_export=/tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:15:30.338 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:15:30.338 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3 /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export 2024-09-17T23:15:30.855 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-17T23:15:30.866 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3 /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:15:31.599 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-17T23:15:31.608 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ cmp /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:15:31.855 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:15:31.976 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:15:31.976 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ snap_name=snap3-clean 2024-09-17T23:15:31.976 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ compare_image_snaps mirror test snap3-clean 2024-09-17T23:15:31.976 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:15:31.977 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:15:31.977 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap3-clean 2024-09-17T23:15:31.977 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local ret=0 2024-09-17T23:15:31.977 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local rmt_export=/tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export 2024-09-17T23:15:31.977 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local loc_export=/tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:15:31.977 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:15:31.977 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3-clean /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export 2024-09-17T23:15:32.480 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-17T23:15:32.490 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3-clean /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:15:33.184 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-17T23:15:33.193 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ cmp /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:15:33.439 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:15:33.561 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:15:33.561 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 10` 2024-09-17T23:15:33.561 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ snap_name=snap4 2024-09-17T23:15:33.561 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ compare_image_snaps mirror test snap4 2024-09-17T23:15:33.561 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:15:33.561 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:15:33.561 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap4 2024-09-17T23:15:33.561 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local ret=0 2024-09-17T23:15:33.561 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local rmt_export=/tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export 2024-09-17T23:15:33.561 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local loc_export=/tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:15:33.561 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:15:33.562 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4 /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export 2024-09-17T23:15:34.075 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-17T23:15:34.084 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4 /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:15:34.803 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-17T23:15:34.811 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ cmp /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:15:35.057 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:15:35.178 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:15:35.178 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ snap_name=snap4-clean 2024-09-17T23:15:35.178 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ compare_image_snaps mirror test snap4-clean 2024-09-17T23:15:35.178 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:15:35.178 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:15:35.178 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap4-clean 2024-09-17T23:15:35.179 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local ret=0 2024-09-17T23:15:35.179 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local rmt_export=/tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export 2024-09-17T23:15:35.179 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local loc_export=/tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:15:35.179 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:15:35.179 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4-clean /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export 2024-09-17T23:15:35.687 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-17T23:15:35.698 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4-clean /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:15:36.426 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-17T23:15:36.433 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ cmp /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:15:36.686 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:15:36.810 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:15:36.810 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 10` 2024-09-17T23:15:36.810 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ snap_name=snap5 2024-09-17T23:15:36.810 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ compare_image_snaps mirror test snap5 2024-09-17T23:15:36.810 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:15:36.810 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:15:36.810 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap5 2024-09-17T23:15:36.810 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local ret=0 2024-09-17T23:15:36.810 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local rmt_export=/tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export 2024-09-17T23:15:36.810 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local loc_export=/tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:15:36.810 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:15:36.811 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5 /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export 2024-09-17T23:15:37.339 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-17T23:15:37.350 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5 /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:15:37.973 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-17T23:15:37.981 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ cmp /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:15:38.229 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:15:38.349 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:15:38.350 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ snap_name=snap5-clean 2024-09-17T23:15:38.350 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ compare_image_snaps mirror test snap5-clean 2024-09-17T23:15:38.350 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:15:38.350 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:15:38.350 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap5-clean 2024-09-17T23:15:38.350 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local ret=0 2024-09-17T23:15:38.350 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local rmt_export=/tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export 2024-09-17T23:15:38.350 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local loc_export=/tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:15:38.350 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:15:38.350 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5-clean /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export 2024-09-17T23:15:38.865 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-17T23:15:38.876 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5-clean /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:15:38.908 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-09-17T23:15:38.909 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-09-17T23:15:38.909 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2024-09-17T23:15:38.910 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi038.stderr:2024-09-17T23:15:38.907+0000 7f4d7c1be640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 40684) UID: 1000 2024-09-17T23:15:39.559 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-17T23:15:39.571 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ cmp /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:15:39.819 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:15:39.942 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:15:39.942 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 10` 2024-09-17T23:15:39.943 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ snap_name=snap6 2024-09-17T23:15:39.943 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ compare_image_snaps mirror test snap6 2024-09-17T23:15:39.943 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:15:39.943 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:15:39.943 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap6 2024-09-17T23:15:39.943 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local ret=0 2024-09-17T23:15:39.943 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local rmt_export=/tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export 2024-09-17T23:15:39.943 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local loc_export=/tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:15:39.943 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:15:39.943 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6 /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export 2024-09-17T23:15:40.451 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-17T23:15:40.461 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6 /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:15:41.096 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-17T23:15:41.107 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ cmp /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:15:41.354 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:15:41.477 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:15:41.477 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ snap_name=snap6-clean 2024-09-17T23:15:41.477 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ compare_image_snaps mirror test snap6-clean 2024-09-17T23:15:41.477 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:15:41.477 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:15:41.477 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap6-clean 2024-09-17T23:15:41.477 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local ret=0 2024-09-17T23:15:41.477 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local rmt_export=/tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export 2024-09-17T23:15:41.477 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local loc_export=/tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:15:41.477 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:15:41.477 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6-clean /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export 2024-09-17T23:15:41.992 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-17T23:15:42.004 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6-clean /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:15:42.761 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-17T23:15:42.770 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ cmp /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:15:43.017 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:15:43.138 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:15:43.138 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 10` 2024-09-17T23:15:43.138 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ snap_name=snap7 2024-09-17T23:15:43.138 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ compare_image_snaps mirror test snap7 2024-09-17T23:15:43.138 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:15:43.138 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:15:43.138 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap7 2024-09-17T23:15:43.138 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local ret=0 2024-09-17T23:15:43.138 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local rmt_export=/tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export 2024-09-17T23:15:43.138 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local loc_export=/tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:15:43.138 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:15:43.139 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7 /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export 2024-09-17T23:15:43.653 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-17T23:15:43.664 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7 /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:15:44.418 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-17T23:15:44.428 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ cmp /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:15:44.678 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:15:44.803 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:15:44.804 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ snap_name=snap7-clean 2024-09-17T23:15:44.804 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ compare_image_snaps mirror test snap7-clean 2024-09-17T23:15:44.804 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:15:44.804 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:15:44.804 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap7-clean 2024-09-17T23:15:44.804 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local ret=0 2024-09-17T23:15:44.804 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local rmt_export=/tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export 2024-09-17T23:15:44.804 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local loc_export=/tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:15:44.804 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:15:44.804 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7-clean /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export 2024-09-17T23:15:45.334 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-17T23:15:45.344 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7-clean /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:15:45.910 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-09-17T23:15:45.910 INFO:teuthology.orchestra.run:waiting for 600 2024-09-17T23:15:45.911 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-09-17T23:15:45.911 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-09-17T23:15:45.911 DEBUG:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-09-17T23:15:45.914 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-09-17T23:15:45.914 INFO:tasks.rbd_mirror_thrash:waiting for 117 secs before thrashing 2024-09-17T23:15:46.102 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-17T23:15:46.112 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ cmp /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:15:46.362 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:15:46.484 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:15:46.485 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 10` 2024-09-17T23:15:46.485 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ snap_name=snap8 2024-09-17T23:15:46.485 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ compare_image_snaps mirror test snap8 2024-09-17T23:15:46.485 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:15:46.485 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:15:46.485 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap8 2024-09-17T23:15:46.485 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local ret=0 2024-09-17T23:15:46.485 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local rmt_export=/tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export 2024-09-17T23:15:46.485 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local loc_export=/tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:15:46.485 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:15:46.485 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8 /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export 2024-09-17T23:15:46.993 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-17T23:15:47.006 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8 /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:15:47.676 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-17T23:15:47.686 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ cmp /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:15:47.932 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:15:48.057 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:15:48.058 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ snap_name=snap8-clean 2024-09-17T23:15:48.058 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ compare_image_snaps mirror test snap8-clean 2024-09-17T23:15:48.058 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:15:48.058 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:15:48.058 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap8-clean 2024-09-17T23:15:48.058 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local ret=0 2024-09-17T23:15:48.058 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local rmt_export=/tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export 2024-09-17T23:15:48.058 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local loc_export=/tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:15:48.058 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:15:48.058 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8-clean /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export 2024-09-17T23:15:48.592 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-17T23:15:48.602 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8-clean /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:15:49.218 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-17T23:15:49.226 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ cmp /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:15:49.472 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:15:49.597 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:15:49.597 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 10` 2024-09-17T23:15:49.597 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ snap_name=snap9 2024-09-17T23:15:49.597 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ compare_image_snaps mirror test snap9 2024-09-17T23:15:49.597 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:15:49.597 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:15:49.598 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap9 2024-09-17T23:15:49.598 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local ret=0 2024-09-17T23:15:49.598 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local rmt_export=/tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export 2024-09-17T23:15:49.598 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local loc_export=/tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:15:49.598 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:15:49.598 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9 /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export 2024-09-17T23:15:50.115 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-17T23:15:50.126 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9 /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:15:50.829 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-17T23:15:50.838 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ cmp /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:15:51.087 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:15:51.209 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:15:51.209 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ snap_name=snap9-clean 2024-09-17T23:15:51.209 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ compare_image_snaps mirror test snap9-clean 2024-09-17T23:15:51.209 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:15:51.209 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:15:51.209 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap9-clean 2024-09-17T23:15:51.209 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local ret=0 2024-09-17T23:15:51.210 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local rmt_export=/tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export 2024-09-17T23:15:51.210 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local loc_export=/tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:15:51.210 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:15:51.210 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9-clean /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export 2024-09-17T23:15:51.744 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-17T23:15:51.756 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9-clean /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:15:52.449 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-17T23:15:52.460 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ cmp /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:15:52.708 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:15:52.832 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:15:52.832 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 10` 2024-09-17T23:15:52.833 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ snap_name=snap10 2024-09-17T23:15:52.833 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ compare_image_snaps mirror test snap10 2024-09-17T23:15:52.833 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:15:52.833 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:15:52.833 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap10 2024-09-17T23:15:52.833 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local ret=0 2024-09-17T23:15:52.833 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local rmt_export=/tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export 2024-09-17T23:15:52.833 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local loc_export=/tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:15:52.833 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:15:52.834 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10 /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export 2024-09-17T23:15:53.357 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-17T23:15:53.369 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10 /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:15:54.107 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-17T23:15:54.116 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ cmp /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:15:54.365 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:15:54.489 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:15:54.489 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ snap_name=snap10-clean 2024-09-17T23:15:54.490 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ compare_image_snaps mirror test snap10-clean 2024-09-17T23:15:54.490 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:15:54.490 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:15:54.490 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap10-clean 2024-09-17T23:15:54.490 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local ret=0 2024-09-17T23:15:54.490 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local rmt_export=/tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export 2024-09-17T23:15:54.490 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local loc_export=/tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:15:54.490 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:15:54.490 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10-clean /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export 2024-09-17T23:15:55.021 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-17T23:15:55.033 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10-clean /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:15:55.759 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-17T23:15:55.771 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ cmp /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:15:56.018 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-test.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-test.export 2024-09-17T23:15:56.140 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:15:56.140 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ seq 1 10 2024-09-17T23:15:56.141 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 10` 2024-09-17T23:15:56.141 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ snap_name=snap1 2024-09-17T23:15:56.141 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_snapshot cluster2 mirror test snap1 2024-09-17T23:15:56.142 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:15:56.142 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:15:56.142 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:15:56.142 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap=snap1 2024-09-17T23:15:56.142 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap1 2024-09-17T23:15:57.098 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing snap: 100% complete...done. 2024-09-17T23:15:57.123 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ snap_name=snap1-clean 2024-09-17T23:15:57.123 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_snapshot cluster2 mirror test snap1-clean 2024-09-17T23:15:57.123 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:15:57.123 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:15:57.123 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:15:57.123 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap=snap1-clean 2024-09-17T23:15:57.123 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap1-clean 2024-09-17T23:15:58.091 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing snap: 100% complete...done. 2024-09-17T23:15:58.125 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 10` 2024-09-17T23:15:58.126 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ snap_name=snap2 2024-09-17T23:15:58.126 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_snapshot cluster2 mirror test snap2 2024-09-17T23:15:58.126 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:15:58.126 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:15:58.126 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:15:58.126 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap=snap2 2024-09-17T23:15:58.126 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap2 2024-09-17T23:15:59.095 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing snap: 100% complete...done. 2024-09-17T23:15:59.122 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ snap_name=snap2-clean 2024-09-17T23:15:59.122 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_snapshot cluster2 mirror test snap2-clean 2024-09-17T23:15:59.122 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:15:59.122 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:15:59.122 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:15:59.122 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap=snap2-clean 2024-09-17T23:15:59.122 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap2-clean 2024-09-17T23:16:00.118 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing snap: 100% complete...done. 2024-09-17T23:16:00.154 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 10` 2024-09-17T23:16:00.154 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ snap_name=snap3 2024-09-17T23:16:00.154 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_snapshot cluster2 mirror test snap3 2024-09-17T23:16:00.154 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:16:00.154 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:16:00.155 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:16:00.155 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap=snap3 2024-09-17T23:16:00.155 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap3 2024-09-17T23:16:01.105 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing snap: 100% complete...done. 2024-09-17T23:16:01.132 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ snap_name=snap3-clean 2024-09-17T23:16:01.132 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_snapshot cluster2 mirror test snap3-clean 2024-09-17T23:16:01.132 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:16:01.132 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:16:01.132 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:16:01.133 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap=snap3-clean 2024-09-17T23:16:01.133 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap3-clean 2024-09-17T23:16:02.122 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing snap: 100% complete...done. 2024-09-17T23:16:02.142 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 10` 2024-09-17T23:16:02.142 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ snap_name=snap4 2024-09-17T23:16:02.142 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_snapshot cluster2 mirror test snap4 2024-09-17T23:16:02.142 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:16:02.142 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:16:02.142 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:16:02.142 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap=snap4 2024-09-17T23:16:02.142 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap4 2024-09-17T23:16:03.137 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing snap: 100% complete...done. 2024-09-17T23:16:03.169 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ snap_name=snap4-clean 2024-09-17T23:16:03.169 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_snapshot cluster2 mirror test snap4-clean 2024-09-17T23:16:03.169 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:16:03.169 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:16:03.169 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:16:03.169 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap=snap4-clean 2024-09-17T23:16:03.169 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap4-clean 2024-09-17T23:16:04.129 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing snap: 100% complete...done. 2024-09-17T23:16:04.162 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 10` 2024-09-17T23:16:04.162 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ snap_name=snap5 2024-09-17T23:16:04.162 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_snapshot cluster2 mirror test snap5 2024-09-17T23:16:04.162 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:16:04.162 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:16:04.162 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:16:04.162 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap=snap5 2024-09-17T23:16:04.163 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap5 2024-09-17T23:16:05.147 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing snap: 100% complete...done. 2024-09-17T23:16:05.171 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ snap_name=snap5-clean 2024-09-17T23:16:05.171 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_snapshot cluster2 mirror test snap5-clean 2024-09-17T23:16:05.171 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:16:05.171 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:16:05.171 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:16:05.172 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap=snap5-clean 2024-09-17T23:16:05.172 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap5-clean 2024-09-17T23:16:06.149 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing snap: 100% complete...done. 2024-09-17T23:16:06.175 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 10` 2024-09-17T23:16:06.175 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ snap_name=snap6 2024-09-17T23:16:06.175 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_snapshot cluster2 mirror test snap6 2024-09-17T23:16:06.176 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:16:06.176 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:16:06.176 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:16:06.176 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap=snap6 2024-09-17T23:16:06.176 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap6 2024-09-17T23:16:07.043 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing snap: 100% complete...done. 2024-09-17T23:16:07.068 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ snap_name=snap6-clean 2024-09-17T23:16:07.068 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_snapshot cluster2 mirror test snap6-clean 2024-09-17T23:16:07.068 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:16:07.069 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:16:07.069 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:16:07.069 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap=snap6-clean 2024-09-17T23:16:07.069 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap6-clean 2024-09-17T23:16:08.167 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing snap: 100% complete...done. 2024-09-17T23:16:08.211 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 10` 2024-09-17T23:16:08.211 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ snap_name=snap7 2024-09-17T23:16:08.211 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_snapshot cluster2 mirror test snap7 2024-09-17T23:16:08.211 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:16:08.211 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:16:08.212 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:16:08.212 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap=snap7 2024-09-17T23:16:08.212 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap7 2024-09-17T23:16:09.166 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing snap: 100% complete...done. 2024-09-17T23:16:09.193 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ snap_name=snap7-clean 2024-09-17T23:16:09.194 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_snapshot cluster2 mirror test snap7-clean 2024-09-17T23:16:09.194 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:16:09.194 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:16:09.194 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:16:09.194 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap=snap7-clean 2024-09-17T23:16:09.194 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap7-clean 2024-09-17T23:16:10.162 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing snap: 100% complete...done. 2024-09-17T23:16:10.192 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 10` 2024-09-17T23:16:10.192 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ snap_name=snap8 2024-09-17T23:16:10.192 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_snapshot cluster2 mirror test snap8 2024-09-17T23:16:10.192 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:16:10.192 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:16:10.192 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:16:10.192 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap=snap8 2024-09-17T23:16:10.192 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap8 2024-09-17T23:16:11.169 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing snap: 100% complete...done. 2024-09-17T23:16:11.192 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ snap_name=snap8-clean 2024-09-17T23:16:11.192 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_snapshot cluster2 mirror test snap8-clean 2024-09-17T23:16:11.192 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:16:11.192 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:16:11.192 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:16:11.192 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap=snap8-clean 2024-09-17T23:16:11.192 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap8-clean 2024-09-17T23:16:12.055 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing snap: 100% complete...done. 2024-09-17T23:16:12.081 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 10` 2024-09-17T23:16:12.081 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ snap_name=snap9 2024-09-17T23:16:12.081 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_snapshot cluster2 mirror test snap9 2024-09-17T23:16:12.081 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:16:12.081 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:16:12.081 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:16:12.082 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap=snap9 2024-09-17T23:16:12.082 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap9 2024-09-17T23:16:13.177 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing snap: 100% complete...done. 2024-09-17T23:16:13.199 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ snap_name=snap9-clean 2024-09-17T23:16:13.199 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_snapshot cluster2 mirror test snap9-clean 2024-09-17T23:16:13.200 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:16:13.200 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:16:13.200 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:16:13.200 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap=snap9-clean 2024-09-17T23:16:13.200 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap9-clean 2024-09-17T23:16:14.187 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing snap: 100% complete...done. 2024-09-17T23:16:14.214 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 10` 2024-09-17T23:16:14.214 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ snap_name=snap10 2024-09-17T23:16:14.214 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_snapshot cluster2 mirror test snap10 2024-09-17T23:16:14.214 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:16:14.214 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:16:14.215 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:16:14.215 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap=snap10 2024-09-17T23:16:14.215 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap10 2024-09-17T23:16:15.185 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing snap: 100% complete...done. 2024-09-17T23:16:15.211 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ snap_name=snap10-clean 2024-09-17T23:16:15.211 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_snapshot cluster2 mirror test snap10-clean 2024-09-17T23:16:15.211 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:16:15.211 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:16:15.211 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:16:15.211 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap=snap10-clean 2024-09-17T23:16:15.212 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap10-clean 2024-09-17T23:16:16.192 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing snap: 100% complete...done. 2024-09-17T23:16:16.215 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image_retry cluster2 mirror test 2024-09-17T23:16:16.215 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:16:16.215 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:16:16.215 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:16:16.215 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-17T23:16:16.215 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0 2024-09-17T23:16:16.216 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image cluster2 mirror test 2024-09-17T23:16:16.216 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:16:16.216 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:16:16.216 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:16:16.216 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 snap purge mirror/test 2024-09-17T23:16:16.264 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 rm mirror/test 2024-09-17T23:16:16.376 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:16:16.373+0000 7ff50242c640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55d142aa3860 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-17T23:16:16.376 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:16:16.373+0000 7ff508c39640 -1 rbd::mirror::ImageReplayer: 0x55d14318a280 [3/d6dfe272-2e04-4592-a96a-96648936ad18] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-17T23:16:18.245 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-17T23:16:18.249 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:16:18.249 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_present cluster1 mirror test deleted 2024-09-17T23:16:18.249 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:16:18.249 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:16:18.249 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:16:18.249 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=deleted 2024-09-17T23:16:18.249 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:16:18.249 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:16:18.250 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:16:18.250 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror test 2024-09-17T23:16:18.250 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:16:18.250 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:16:18.250 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=test 2024-09-17T23:16:18.250 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-09-17T23:16:18.250 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:16:18.283 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image test: (2) No such file or directory 2024-09-17T23:16:18.286 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image_id= 2024-09-17T23:16:18.286 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-17T23:16:18.286 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.1 2024-09-17T23:16:18.388 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_present cluster1 mirror test deleted '' 2024-09-17T23:16:18.388 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:16:18.388 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:16:18.389 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=test 2024-09-17T23:16:18.389 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=deleted 2024-09-17T23:16:18.389 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:16:18.389 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=deleted 2024-09-17T23:16:18.389 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_image_id 2024-09-17T23:16:18.389 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror test 2024-09-17T23:16:18.389 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:16:18.389 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:16:18.389 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=test 2024-09-17T23:16:18.389 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-09-17T23:16:18.389 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:16:18.421 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image test: (2) No such file or directory 2024-09-17T23:16:18.424 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_image_id= 2024-09-17T23:16:18.424 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:16:18.425 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test deleted = deleted 2024-09-17T23:16:18.425 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:16:18.425 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ testlog 'TEST: create many images' 2024-09-17T23:16:18.425 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ tee -a /tmp/tmp.s5MEFDgsej/rbd-mirror.test.log 2024-09-17T23:16:18.425 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ date '+%F %T' 2024-09-17T23:16:18.425 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ echo 2024-09-17 23:16:18 TEST: create many images 2024-09-17T23:16:18.426 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:2024-09-17 23:16:18 TEST: create many images 2024-09-17T23:16:18.426 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ snap_name=snap 2024-09-17T23:16:18.426 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ seq 1 50 2024-09-17T23:16:18.428 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:16:18.428 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_1 2024-09-17T23:16:18.428 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_image_and_enable_mirror cluster2 mirror image_1 snapshot 128M 2024-09-17T23:16:18.428 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:16:18.428 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:16:18.428 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:16:18.428 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:16:18.428 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_1 2024-09-17T23:16:18.428 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:16:18.429 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=snapshot 2024-09-17T23:16:18.429 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n snapshot ']' 2024-09-17T23:16:18.429 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:16:18.429 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_image cluster2 mirror image_1 128M 2024-09-17T23:16:18.429 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:16:18.429 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:16:18.429 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:16:18.429 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:16:18.429 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_1 2024-09-17T23:16:18.429 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:16:18.429 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local size=128 2024-09-17T23:16:18.429 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n 128M ']' 2024-09-17T23:16:18.429 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ size=128M 2024-09-17T23:16:18.430 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:16:18.430 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_1 2024-09-17T23:16:18.475 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ is_pool_mirror_mode_image mirror 2024-09-17T23:16:18.475 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:16:18.475 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' image = image ']' 2024-09-17T23:16:18.475 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:16:18.475 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ enable_mirror cluster2 mirror image_1 snapshot 2024-09-17T23:16:18.476 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:16:18.476 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:16:18.476 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_1 2024-09-17T23:16:18.476 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=snapshot 2024-09-17T23:16:18.476 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_1 snapshot 2024-09-17T23:16:19.062 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Mirroring enabled 2024-09-17T23:16:19.070 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 info mirror/image_1 2024-09-17T23:16:19.109 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'image_1': 2024-09-17T23:16:19.110 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 128 MiB in 32 objects 2024-09-17T23:16:19.110 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:16:19.110 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 1 2024-09-17T23:16:19.110 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 127689cef835 2024-09-17T23:16:19.110 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.127689cef835 2024-09-17T23:16:19.110 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:16:19.110 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-17T23:16:19.110 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:16:19.110 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:16:19.110 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:16:18 2024 2024-09-17T23:16:19.110 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:16:18 2024 2024-09-17T23:16:19.110 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:16:18 2024 2024-09-17T23:16:19.111 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:16:19.111 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:16:19.111 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: 5ca82a92-11a6-461d-96fa-d42ff94d4c65 2024-09-17T23:16:19.111 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: true 2024-09-17T23:16:19.113 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n '' ']' 2024-09-17T23:16:19.113 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ stress_write_image cluster2 mirror image_1 2024-09-17T23:16:19.113 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:16:19.113 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:16:19.113 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_1 2024-09-17T23:16:19.113 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-17T23:16:19.115 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local duration=7 2024-09-17T23:16:19.115 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set +e 2024-09-17T23:16:19.115 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_1 --debug-rbd=20 --debug-journaler=20 2024-09-17T23:16:19.266 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:100 IOs, 18127872 bytes 2024-09-17T23:16:19.353 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:200 IOs, 50220032 bytes 2024-09-17T23:16:19.458 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:300 IOs, 98272256 bytes 2024-09-17T23:16:19.574 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:400 IOs, 159217664 bytes 2024-09-17T23:16:19.701 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:500 IOs, 232121344 bytes 2024-09-17T23:16:19.781 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:600 IOs, 250957824 bytes 2024-09-17T23:16:19.923 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:700 IOs, 344455168 bytes 2024-09-17T23:16:20.045 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:800 IOs, 400697344 bytes 2024-09-17T23:16:20.154 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:900 IOs, 447181824 bytes 2024-09-17T23:16:20.258 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1000 IOs, 492653568 bytes 2024-09-17T23:16:20.402 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1100 IOs, 580902912 bytes 2024-09-17T23:16:20.537 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1200 IOs, 637774848 bytes 2024-09-17T23:16:20.645 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1300 IOs, 688591872 bytes 2024-09-17T23:16:20.740 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1400 IOs, 721512448 bytes 2024-09-17T23:16:20.802 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1500 IOs, 722141184 bytes 2024-09-17T23:16:20.814 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.4808481220990549) > weight (0.25) 2024-09-17T23:16:20.814 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.3746549219328601) > weight (0.25) 2024-09-17T23:16:20.814 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.5478413463875821) > weight (0.25) 2024-09-17T23:16:20.814 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.8788574750865095) > weight (0.25) 2024-09-17T23:16:20.814 INFO:tasks.rbd_mirror_thrash:waiting for 116 secs before thrashing 2024-09-17T23:16:20.875 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1600 IOs, 730245120 bytes 2024-09-17T23:16:20.942 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1700 IOs, 738682880 bytes 2024-09-17T23:16:21.086 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1800 IOs, 813331456 bytes 2024-09-17T23:16:21.155 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1900 IOs, 817882112 bytes 2024-09-17T23:16:21.293 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2000 IOs, 898257920 bytes 2024-09-17T23:16:21.396 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2100 IOs, 942068736 bytes 2024-09-17T23:16:21.488 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2200 IOs, 969033728 bytes 2024-09-17T23:16:21.576 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2300 IOs, 993002496 bytes 2024-09-17T23:16:21.735 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2400 IOs, 1128083456 bytes 2024-09-17T23:16:21.849 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2500 IOs, 1153747968 bytes 2024-09-17T23:16:21.951 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2600 IOs, 1201805312 bytes 2024-09-17T23:16:22.079 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2700 IOs, 1250066432 bytes 2024-09-17T23:16:22.194 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2800 IOs, 1307548672 bytes 2024-09-17T23:16:22.274 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2900 IOs, 1322149888 bytes 2024-09-17T23:16:22.375 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3000 IOs, 1358270464 bytes 2024-09-17T23:16:22.491 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3100 IOs, 1413656576 bytes 2024-09-17T23:16:22.576 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3200 IOs, 1438936064 bytes 2024-09-17T23:16:22.657 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3300 IOs, 1456344064 bytes 2024-09-17T23:16:22.751 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3400 IOs, 1490779136 bytes 2024-09-17T23:16:22.880 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3500 IOs, 1560280064 bytes 2024-09-17T23:16:22.994 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3600 IOs, 1613602816 bytes 2024-09-17T23:16:23.069 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3700 IOs, 1630867456 bytes 2024-09-17T23:16:23.209 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3800 IOs, 1707031552 bytes 2024-09-17T23:16:23.310 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3900 IOs, 1748326400 bytes 2024-09-17T23:16:23.420 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4000 IOs, 1793718272 bytes 2024-09-17T23:16:23.505 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4100 IOs, 1821422592 bytes 2024-09-17T23:16:23.628 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4200 IOs, 1885466624 bytes 2024-09-17T23:16:23.734 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4300 IOs, 1924536320 bytes 2024-09-17T23:16:23.851 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4400 IOs, 1988423680 bytes 2024-09-17T23:16:23.965 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4500 IOs, 2042255360 bytes 2024-09-17T23:16:24.104 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4600 IOs, 2126853120 bytes 2024-09-17T23:16:24.199 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4700 IOs, 2157696000 bytes 2024-09-17T23:16:24.286 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4800 IOs, 2180402176 bytes 2024-09-17T23:16:24.379 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4900 IOs, 2213820416 bytes 2024-09-17T23:16:24.451 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5000 IOs, 2224471040 bytes 2024-09-17T23:16:24.546 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5100 IOs, 2257556480 bytes 2024-09-17T23:16:24.712 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5200 IOs, 2367778816 bytes 2024-09-17T23:16:24.845 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5300 IOs, 2445996032 bytes 2024-09-17T23:16:24.937 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5400 IOs, 2477186048 bytes 2024-09-17T23:16:25.055 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5500 IOs, 2531560448 bytes 2024-09-17T23:16:25.146 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5600 IOs, 2562912256 bytes 2024-09-17T23:16:25.252 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5700 IOs, 2608884736 bytes 2024-09-17T23:16:25.376 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5800 IOs, 2665463808 bytes 2024-09-17T23:16:25.457 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5900 IOs, 2704760832 bytes 2024-09-17T23:16:25.585 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6000 IOs, 2767690752 bytes 2024-09-17T23:16:25.712 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6100 IOs, 2811527168 bytes 2024-09-17T23:16:25.791 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6200 IOs, 2833625088 bytes 2024-09-17T23:16:25.904 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6300 IOs, 2877608960 bytes 2024-09-17T23:16:26.020 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6400 IOs, 2951173120 bytes 2024-09-17T23:16:26.124 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ error_code=124 2024-09-17T23:16:26.124 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set -e 2024-09-17T23:16:26.124 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' 124 -eq 124 ']' 2024-09-17T23:16:26.124 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:16:26.124 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:16:26.124 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_2 2024-09-17T23:16:26.124 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_image_and_enable_mirror cluster2 mirror image_2 snapshot 128M 2024-09-17T23:16:26.124 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:16:26.125 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:16:26.125 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:16:26.125 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:16:26.125 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_2 2024-09-17T23:16:26.125 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:16:26.125 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=snapshot 2024-09-17T23:16:26.125 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n snapshot ']' 2024-09-17T23:16:26.125 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:16:26.125 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_image cluster2 mirror image_2 128M 2024-09-17T23:16:26.125 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:16:26.125 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:16:26.125 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:16:26.125 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:16:26.125 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_2 2024-09-17T23:16:26.125 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:16:26.126 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local size=128 2024-09-17T23:16:26.126 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n 128M ']' 2024-09-17T23:16:26.126 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ size=128M 2024-09-17T23:16:26.126 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:16:26.126 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_2 2024-09-17T23:16:26.173 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ is_pool_mirror_mode_image mirror 2024-09-17T23:16:26.173 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:16:26.173 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' image = image ']' 2024-09-17T23:16:26.173 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:16:26.173 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ enable_mirror cluster2 mirror image_2 snapshot 2024-09-17T23:16:26.173 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:16:26.173 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:16:26.173 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_2 2024-09-17T23:16:26.174 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=snapshot 2024-09-17T23:16:26.174 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_2 snapshot 2024-09-17T23:16:27.064 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Mirroring enabled 2024-09-17T23:16:27.073 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 info mirror/image_2 2024-09-17T23:16:27.113 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'image_2': 2024-09-17T23:16:27.113 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 128 MiB in 32 objects 2024-09-17T23:16:27.113 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:16:27.114 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 1 2024-09-17T23:16:27.114 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 12822542ba08 2024-09-17T23:16:27.114 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.12822542ba08 2024-09-17T23:16:27.114 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:16:27.114 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-17T23:16:27.114 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:16:27.114 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:16:27.114 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:16:26 2024 2024-09-17T23:16:27.114 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:16:26 2024 2024-09-17T23:16:27.114 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:16:26 2024 2024-09-17T23:16:27.114 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:16:27.114 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:16:27.114 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: e4119d6f-e55a-46f9-97e1-502365b9e208 2024-09-17T23:16:27.115 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: true 2024-09-17T23:16:27.117 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n '' ']' 2024-09-17T23:16:27.117 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ stress_write_image cluster2 mirror image_2 2024-09-17T23:16:27.117 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:16:27.117 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:16:27.118 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_2 2024-09-17T23:16:27.118 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-17T23:16:27.119 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local duration=14 2024-09-17T23:16:27.119 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set +e 2024-09-17T23:16:27.119 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_2 --debug-rbd=20 --debug-journaler=20 2024-09-17T23:16:27.305 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:100 IOs, 60131328 bytes 2024-09-17T23:16:27.371 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:200 IOs, 75205632 bytes 2024-09-17T23:16:27.432 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:300 IOs, 76759040 bytes 2024-09-17T23:16:27.553 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:400 IOs, 144546816 bytes 2024-09-17T23:16:27.673 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:500 IOs, 215121920 bytes 2024-09-17T23:16:27.815 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:600 IOs, 306932736 bytes 2024-09-17T23:16:27.889 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:700 IOs, 323835904 bytes 2024-09-17T23:16:27.983 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:800 IOs, 364493824 bytes 2024-09-17T23:16:28.076 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:900 IOs, 400360448 bytes 2024-09-17T23:16:28.176 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1000 IOs, 451298304 bytes 2024-09-17T23:16:28.289 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1100 IOs, 500825088 bytes 2024-09-17T23:16:28.345 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1200 IOs, 501457920 bytes 2024-09-17T23:16:28.461 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1300 IOs, 568395776 bytes 2024-09-17T23:16:28.549 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1400 IOs, 603068416 bytes 2024-09-17T23:16:28.609 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1500 IOs, 603682816 bytes 2024-09-17T23:16:28.686 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1600 IOs, 620929024 bytes 2024-09-17T23:16:28.769 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1700 IOs, 654717952 bytes 2024-09-17T23:16:28.924 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1800 IOs, 761884672 bytes 2024-09-17T23:16:29.009 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1900 IOs, 777441280 bytes 2024-09-17T23:16:29.079 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2000 IOs, 785202176 bytes 2024-09-17T23:16:29.211 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2100 IOs, 878911488 bytes 2024-09-17T23:16:29.340 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2200 IOs, 943631360 bytes 2024-09-17T23:16:29.421 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2300 IOs, 972020736 bytes 2024-09-17T23:16:29.503 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2400 IOs, 1012088832 bytes 2024-09-17T23:16:29.613 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2500 IOs, 1043005440 bytes 2024-09-17T23:16:29.701 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2600 IOs, 1073053696 bytes 2024-09-17T23:16:29.787 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2700 IOs, 1081783296 bytes 2024-09-17T23:16:29.891 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2800 IOs, 1146603520 bytes 2024-09-17T23:16:30.022 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2900 IOs, 1208117248 bytes 2024-09-17T23:16:30.096 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3000 IOs, 1220240384 bytes 2024-09-17T23:16:30.169 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3100 IOs, 1231013888 bytes 2024-09-17T23:16:30.353 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3200 IOs, 1360130048 bytes 2024-09-17T23:16:30.448 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3300 IOs, 1407208448 bytes 2024-09-17T23:16:30.552 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3400 IOs, 1448005632 bytes 2024-09-17T23:16:30.701 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3500 IOs, 1549153280 bytes 2024-09-17T23:16:30.781 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3600 IOs, 1574836224 bytes 2024-09-17T23:16:30.868 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3700 IOs, 1605349376 bytes 2024-09-17T23:16:30.966 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3800 IOs, 1658809344 bytes 2024-09-17T23:16:31.133 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3900 IOs, 1760172032 bytes 2024-09-17T23:16:31.243 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4000 IOs, 1813420032 bytes 2024-09-17T23:16:31.340 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4100 IOs, 1853853696 bytes 2024-09-17T23:16:31.424 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4200 IOs, 1883926528 bytes 2024-09-17T23:16:31.530 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4300 IOs, 1930560512 bytes 2024-09-17T23:16:31.627 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4400 IOs, 1965658112 bytes 2024-09-17T23:16:31.710 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4500 IOs, 1990837248 bytes 2024-09-17T23:16:31.827 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4600 IOs, 2054079488 bytes 2024-09-17T23:16:31.930 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4700 IOs, 2100516864 bytes 2024-09-17T23:16:32.010 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4800 IOs, 2121637888 bytes 2024-09-17T23:16:32.095 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4900 IOs, 2148353024 bytes 2024-09-17T23:16:32.204 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5000 IOs, 2206802944 bytes 2024-09-17T23:16:32.341 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5100 IOs, 2293968896 bytes 2024-09-17T23:16:32.444 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5200 IOs, 2334464000 bytes 2024-09-17T23:16:32.545 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5300 IOs, 2378237952 bytes 2024-09-17T23:16:32.607 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5400 IOs, 2378840064 bytes 2024-09-17T23:16:32.732 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5500 IOs, 2449813504 bytes 2024-09-17T23:16:32.850 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5600 IOs, 2505758720 bytes 2024-09-17T23:16:32.947 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5700 IOs, 2545015808 bytes 2024-09-17T23:16:33.013 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5800 IOs, 2562791424 bytes 2024-09-17T23:16:33.105 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5900 IOs, 2579244032 bytes 2024-09-17T23:16:33.191 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6000 IOs, 2613228544 bytes 2024-09-17T23:16:33.283 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6100 IOs, 2643264512 bytes 2024-09-17T23:16:33.371 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6200 IOs, 2668311552 bytes 2024-09-17T23:16:33.484 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6300 IOs, 2721058816 bytes 2024-09-17T23:16:33.555 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6400 IOs, 2732699648 bytes 2024-09-17T23:16:33.665 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6500 IOs, 2785839104 bytes 2024-09-17T23:16:33.765 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6600 IOs, 2828346368 bytes 2024-09-17T23:16:33.905 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6700 IOs, 2910149632 bytes 2024-09-17T23:16:33.987 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6800 IOs, 2931066880 bytes 2024-09-17T23:16:34.099 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6900 IOs, 2985104384 bytes 2024-09-17T23:16:34.163 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7000 IOs, 2988961792 bytes 2024-09-17T23:16:34.232 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7100 IOs, 2994354176 bytes 2024-09-17T23:16:34.308 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7200 IOs, 3011166208 bytes 2024-09-17T23:16:34.394 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7300 IOs, 3028708352 bytes 2024-09-17T23:16:34.525 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7400 IOs, 3103727616 bytes 2024-09-17T23:16:34.603 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7500 IOs, 3120972800 bytes 2024-09-17T23:16:34.688 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7600 IOs, 3148598272 bytes 2024-09-17T23:16:34.808 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7700 IOs, 3210364928 bytes 2024-09-17T23:16:34.895 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7800 IOs, 3236207616 bytes 2024-09-17T23:16:34.998 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7900 IOs, 3279642624 bytes 2024-09-17T23:16:35.088 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8000 IOs, 3308197888 bytes 2024-09-17T23:16:35.199 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8100 IOs, 3375176704 bytes 2024-09-17T23:16:35.335 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8200 IOs, 3458496512 bytes 2024-09-17T23:16:35.454 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8300 IOs, 3503059968 bytes 2024-09-17T23:16:35.515 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8400 IOs, 3503673344 bytes 2024-09-17T23:16:35.651 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8500 IOs, 3577414656 bytes 2024-09-17T23:16:35.772 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8600 IOs, 3640862720 bytes 2024-09-17T23:16:35.865 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8700 IOs, 3683302400 bytes 2024-09-17T23:16:35.961 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8800 IOs, 3706674176 bytes 2024-09-17T23:16:36.091 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8900 IOs, 3776936960 bytes 2024-09-17T23:16:36.166 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9000 IOs, 3798122496 bytes 2024-09-17T23:16:36.308 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9100 IOs, 3878988800 bytes 2024-09-17T23:16:36.394 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9200 IOs, 3905345536 bytes 2024-09-17T23:16:36.513 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9300 IOs, 3968196608 bytes 2024-09-17T23:16:36.606 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9400 IOs, 3996147712 bytes 2024-09-17T23:16:36.672 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9500 IOs, 3996766208 bytes 2024-09-17T23:16:36.791 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9600 IOs, 4056020992 bytes 2024-09-17T23:16:36.919 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9700 IOs, 4126115840 bytes 2024-09-17T23:16:36.999 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9800 IOs, 4142117888 bytes 2024-09-17T23:16:37.130 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9900 IOs, 4210805760 bytes 2024-09-17T23:16:37.192 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10000 IOs, 4211432448 bytes 2024-09-17T23:16:37.260 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10100 IOs, 4212060160 bytes 2024-09-17T23:16:37.343 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10200 IOs, 4233920512 bytes 2024-09-17T23:16:37.463 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10300 IOs, 4292169728 bytes 2024-09-17T23:16:37.555 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10400 IOs, 4320706560 bytes 2024-09-17T23:16:37.665 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10500 IOs, 4365759488 bytes 2024-09-17T23:16:37.749 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10600 IOs, 4391649280 bytes 2024-09-17T23:16:37.817 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10700 IOs, 4415918080 bytes 2024-09-17T23:16:37.947 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10800 IOs, 4481403904 bytes 2024-09-17T23:16:38.051 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10900 IOs, 4550169600 bytes 2024-09-17T23:16:38.155 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:11000 IOs, 4551920640 bytes 2024-09-17T23:16:38.239 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:11100 IOs, 4575188992 bytes 2024-09-17T23:16:38.319 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:11200 IOs, 4591364096 bytes 2024-09-17T23:16:38.386 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:11300 IOs, 4592009216 bytes 2024-09-17T23:16:38.521 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:11400 IOs, 4667454464 bytes 2024-09-17T23:16:38.610 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:11500 IOs, 4719251456 bytes 2024-09-17T23:16:38.765 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:11600 IOs, 4790565888 bytes 2024-09-17T23:16:38.880 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:11700 IOs, 4842635264 bytes 2024-09-17T23:16:38.968 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:11800 IOs, 4866371584 bytes 2024-09-17T23:16:39.049 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:11900 IOs, 4881310720 bytes 2024-09-17T23:16:39.126 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:12000 IOs, 4893849600 bytes 2024-09-17T23:16:39.201 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:12100 IOs, 4906568704 bytes 2024-09-17T23:16:39.288 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:12200 IOs, 4935592960 bytes 2024-09-17T23:16:39.378 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:12300 IOs, 4961520640 bytes 2024-09-17T23:16:39.558 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:12400 IOs, 5086826496 bytes 2024-09-17T23:16:39.637 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:12500 IOs, 5108804608 bytes 2024-09-17T23:16:39.739 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:12600 IOs, 5144998912 bytes 2024-09-17T23:16:39.820 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:12700 IOs, 5159425024 bytes 2024-09-17T23:16:39.905 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:12800 IOs, 5184081920 bytes 2024-09-17T23:16:40.038 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:12900 IOs, 5259761664 bytes 2024-09-17T23:16:40.187 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:13000 IOs, 5344308224 bytes 2024-09-17T23:16:40.273 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:13100 IOs, 5372104704 bytes 2024-09-17T23:16:40.395 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:13200 IOs, 5436098560 bytes 2024-09-17T23:16:40.473 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:13300 IOs, 5442970624 bytes 2024-09-17T23:16:40.566 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:13400 IOs, 5477040128 bytes 2024-09-17T23:16:40.630 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:13500 IOs, 5487227904 bytes 2024-09-17T23:16:40.728 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:13600 IOs, 5513199616 bytes 2024-09-17T23:16:40.828 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:13700 IOs, 5546260480 bytes 2024-09-17T23:16:40.954 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:13800 IOs, 5631171584 bytes 2024-09-17T23:16:41.083 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:13900 IOs, 5686355968 bytes 2024-09-17T23:16:41.124 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ error_code=124 2024-09-17T23:16:41.125 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set -e 2024-09-17T23:16:41.125 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' 124 -eq 124 ']' 2024-09-17T23:16:41.125 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:16:41.125 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:16:41.125 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_3 2024-09-17T23:16:41.125 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_image_and_enable_mirror cluster2 mirror image_3 snapshot 128M 2024-09-17T23:16:41.125 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:16:41.125 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:16:41.125 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:16:41.125 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:16:41.125 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_3 2024-09-17T23:16:41.125 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:16:41.125 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=snapshot 2024-09-17T23:16:41.125 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n snapshot ']' 2024-09-17T23:16:41.126 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:16:41.126 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_image cluster2 mirror image_3 128M 2024-09-17T23:16:41.126 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:16:41.126 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:16:41.126 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:16:41.127 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:16:41.127 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_3 2024-09-17T23:16:41.127 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:16:41.127 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local size=128 2024-09-17T23:16:41.127 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n 128M ']' 2024-09-17T23:16:41.127 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ size=128M 2024-09-17T23:16:41.127 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:16:41.127 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_3 2024-09-17T23:16:41.172 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ is_pool_mirror_mode_image mirror 2024-09-17T23:16:41.172 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:16:41.172 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' image = image ']' 2024-09-17T23:16:41.172 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:16:41.172 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ enable_mirror cluster2 mirror image_3 snapshot 2024-09-17T23:16:41.173 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:16:41.173 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:16:41.173 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_3 2024-09-17T23:16:41.173 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=snapshot 2024-09-17T23:16:41.173 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_3 snapshot 2024-09-17T23:16:42.071 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Mirroring enabled 2024-09-17T23:16:42.080 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 info mirror/image_3 2024-09-17T23:16:42.119 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'image_3': 2024-09-17T23:16:42.120 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 128 MiB in 32 objects 2024-09-17T23:16:42.120 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:16:42.120 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 1 2024-09-17T23:16:42.120 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 128d484beebe 2024-09-17T23:16:42.120 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.128d484beebe 2024-09-17T23:16:42.120 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:16:42.120 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-17T23:16:42.120 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:16:42.120 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:16:42.120 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:16:41 2024 2024-09-17T23:16:42.120 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:16:41 2024 2024-09-17T23:16:42.120 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:16:41 2024 2024-09-17T23:16:42.120 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:16:42.121 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:16:42.121 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: cb923d84-7e5a-4d91-9543-3952a9f3edb7 2024-09-17T23:16:42.121 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: true 2024-09-17T23:16:42.123 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n '' ']' 2024-09-17T23:16:42.123 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ stress_write_image cluster2 mirror image_3 2024-09-17T23:16:42.123 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:16:42.123 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:16:42.123 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_3 2024-09-17T23:16:42.123 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-17T23:16:42.125 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local duration=5 2024-09-17T23:16:42.125 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set +e 2024-09-17T23:16:42.125 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_3 --debug-rbd=20 --debug-journaler=20 2024-09-17T23:16:42.282 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:100 IOs, 29048832 bytes 2024-09-17T23:16:42.384 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:200 IOs, 64743424 bytes 2024-09-17T23:16:42.466 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:300 IOs, 81721344 bytes 2024-09-17T23:16:42.562 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:400 IOs, 117949440 bytes 2024-09-17T23:16:42.681 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:500 IOs, 193120256 bytes 2024-09-17T23:16:42.794 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:600 IOs, 232781824 bytes 2024-09-17T23:16:42.938 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:700 IOs, 320575488 bytes 2024-09-17T23:16:43.051 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:800 IOs, 389904384 bytes 2024-09-17T23:16:43.158 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:900 IOs, 407279616 bytes 2024-09-17T23:16:43.231 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1000 IOs, 432993280 bytes 2024-09-17T23:16:43.348 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1100 IOs, 482736128 bytes 2024-09-17T23:16:43.469 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1200 IOs, 535384064 bytes 2024-09-17T23:16:43.545 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1300 IOs, 539677696 bytes 2024-09-17T23:16:43.678 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1400 IOs, 612946944 bytes 2024-09-17T23:16:43.771 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1500 IOs, 640871424 bytes 2024-09-17T23:16:43.893 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1600 IOs, 706010112 bytes 2024-09-17T23:16:43.960 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1700 IOs, 720841728 bytes 2024-09-17T23:16:44.078 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1800 IOs, 756257792 bytes 2024-09-17T23:16:44.269 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1900 IOs, 893243392 bytes 2024-09-17T23:16:44.355 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2000 IOs, 901441536 bytes 2024-09-17T23:16:44.483 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2100 IOs, 969290752 bytes 2024-09-17T23:16:44.567 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2200 IOs, 982760448 bytes 2024-09-17T23:16:44.646 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2300 IOs, 996370432 bytes 2024-09-17T23:16:44.753 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2400 IOs, 1043271680 bytes 2024-09-17T23:16:44.888 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2500 IOs, 1109813248 bytes 2024-09-17T23:16:44.984 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2600 IOs, 1139508224 bytes 2024-09-17T23:16:45.064 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2700 IOs, 1155936256 bytes 2024-09-17T23:16:45.153 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2800 IOs, 1176280064 bytes 2024-09-17T23:16:45.240 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2900 IOs, 1211615232 bytes 2024-09-17T23:16:45.348 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3000 IOs, 1245767680 bytes 2024-09-17T23:16:45.461 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3100 IOs, 1294040064 bytes 2024-09-17T23:16:45.641 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3200 IOs, 1355332608 bytes 2024-09-17T23:16:45.724 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3300 IOs, 1375227904 bytes 2024-09-17T23:16:45.830 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3400 IOs, 1418470400 bytes 2024-09-17T23:16:45.929 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3500 IOs, 1443048448 bytes 2024-09-17T23:16:46.010 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3600 IOs, 1458404352 bytes 2024-09-17T23:16:46.130 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3700 IOs, 1517940736 bytes 2024-09-17T23:16:46.250 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3800 IOs, 1579040768 bytes 2024-09-17T23:16:46.370 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3900 IOs, 1637369856 bytes 2024-09-17T23:16:46.538 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4000 IOs, 1752644608 bytes 2024-09-17T23:16:46.657 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4100 IOs, 1808659456 bytes 2024-09-17T23:16:46.758 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4200 IOs, 1841714176 bytes 2024-09-17T23:16:46.829 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4300 IOs, 1851521024 bytes 2024-09-17T23:16:46.982 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4400 IOs, 1941777408 bytes 2024-09-17T23:16:47.084 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4500 IOs, 1972629504 bytes 2024-09-17T23:16:47.132 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ error_code=124 2024-09-17T23:16:47.133 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set -e 2024-09-17T23:16:47.133 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' 124 -eq 124 ']' 2024-09-17T23:16:47.133 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:16:47.133 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:16:47.133 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_4 2024-09-17T23:16:47.133 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_image_and_enable_mirror cluster2 mirror image_4 snapshot 128M 2024-09-17T23:16:47.133 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:16:47.133 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:16:47.133 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:16:47.133 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:16:47.133 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_4 2024-09-17T23:16:47.133 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:16:47.135 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=snapshot 2024-09-17T23:16:47.135 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n snapshot ']' 2024-09-17T23:16:47.135 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:16:47.135 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_image cluster2 mirror image_4 128M 2024-09-17T23:16:47.135 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:16:47.135 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:16:47.135 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:16:47.135 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:16:47.135 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_4 2024-09-17T23:16:47.135 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:16:47.135 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local size=128 2024-09-17T23:16:47.136 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n 128M ']' 2024-09-17T23:16:47.136 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ size=128M 2024-09-17T23:16:47.136 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:16:47.136 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_4 2024-09-17T23:16:47.182 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ is_pool_mirror_mode_image mirror 2024-09-17T23:16:47.182 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:16:47.182 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' image = image ']' 2024-09-17T23:16:47.183 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:16:47.183 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ enable_mirror cluster2 mirror image_4 snapshot 2024-09-17T23:16:47.183 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:16:47.183 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:16:47.183 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_4 2024-09-17T23:16:47.183 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=snapshot 2024-09-17T23:16:47.183 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_4 snapshot 2024-09-17T23:16:48.096 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Mirroring enabled 2024-09-17T23:16:48.108 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 info mirror/image_4 2024-09-17T23:16:48.149 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'image_4': 2024-09-17T23:16:48.149 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 128 MiB in 32 objects 2024-09-17T23:16:48.150 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:16:48.150 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 1 2024-09-17T23:16:48.150 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 1299d82760ff 2024-09-17T23:16:48.150 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.1299d82760ff 2024-09-17T23:16:48.150 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:16:48.150 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-17T23:16:48.150 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:16:48.150 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:16:48.150 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:16:47 2024 2024-09-17T23:16:48.150 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:16:47 2024 2024-09-17T23:16:48.150 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:16:47 2024 2024-09-17T23:16:48.150 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:16:48.150 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:16:48.151 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: 1445a68d-ae17-48f5-9088-ad3e0f9e8951 2024-09-17T23:16:48.151 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: true 2024-09-17T23:16:48.153 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n '' ']' 2024-09-17T23:16:48.153 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ stress_write_image cluster2 mirror image_4 2024-09-17T23:16:48.153 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:16:48.153 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:16:48.153 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_4 2024-09-17T23:16:48.153 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-17T23:16:48.155 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local duration=6 2024-09-17T23:16:48.155 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set +e 2024-09-17T23:16:48.155 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_4 --debug-rbd=20 --debug-journaler=20 2024-09-17T23:16:48.328 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:100 IOs, 45403136 bytes 2024-09-17T23:16:48.423 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:200 IOs, 69127168 bytes 2024-09-17T23:16:48.560 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:300 IOs, 153612288 bytes 2024-09-17T23:16:48.631 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:400 IOs, 163599360 bytes 2024-09-17T23:16:48.702 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:500 IOs, 173347840 bytes 2024-09-17T23:16:48.846 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:600 IOs, 259407872 bytes 2024-09-17T23:16:48.916 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:700 IOs, 274264064 bytes 2024-09-17T23:16:49.015 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:800 IOs, 321932288 bytes 2024-09-17T23:16:49.097 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:900 IOs, 341536768 bytes 2024-09-17T23:16:49.188 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1000 IOs, 370522112 bytes 2024-09-17T23:16:49.274 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1100 IOs, 398775296 bytes 2024-09-17T23:16:49.387 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1200 IOs, 457164800 bytes 2024-09-17T23:16:49.454 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1300 IOs, 468703232 bytes 2024-09-17T23:16:49.547 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1400 IOs, 512393216 bytes 2024-09-17T23:16:49.638 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1500 IOs, 550888448 bytes 2024-09-17T23:16:49.732 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1600 IOs, 566711296 bytes 2024-09-17T23:16:49.810 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1700 IOs, 583378944 bytes 2024-09-17T23:16:49.894 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1800 IOs, 610322432 bytes 2024-09-17T23:16:49.963 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1900 IOs, 616863744 bytes 2024-09-17T23:16:50.119 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2000 IOs, 716262400 bytes 2024-09-17T23:16:50.214 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2100 IOs, 746640384 bytes 2024-09-17T23:16:50.301 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2200 IOs, 773939200 bytes 2024-09-17T23:16:50.396 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2300 IOs, 809973760 bytes 2024-09-17T23:16:50.522 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2400 IOs, 882345984 bytes 2024-09-17T23:16:50.633 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2500 IOs, 935519232 bytes 2024-09-17T23:16:50.742 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2600 IOs, 989693952 bytes 2024-09-17T23:16:50.869 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2700 IOs, 1072934912 bytes 2024-09-17T23:16:51.005 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2800 IOs, 1144966144 bytes 2024-09-17T23:16:51.151 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2900 IOs, 1233656832 bytes 2024-09-17T23:16:51.225 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3000 IOs, 1243817984 bytes 2024-09-17T23:16:51.322 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3100 IOs, 1282065408 bytes 2024-09-17T23:16:51.389 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3200 IOs, 1308019712 bytes 2024-09-17T23:16:51.495 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3300 IOs, 1342954496 bytes 2024-09-17T23:16:51.627 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3400 IOs, 1415677952 bytes 2024-09-17T23:16:51.710 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3500 IOs, 1438736384 bytes 2024-09-17T23:16:51.796 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3600 IOs, 1461330944 bytes 2024-09-17T23:16:51.899 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3700 IOs, 1515916288 bytes 2024-09-17T23:16:51.976 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3800 IOs, 1524198400 bytes 2024-09-17T23:16:52.046 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3900 IOs, 1540606976 bytes 2024-09-17T23:16:52.162 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4000 IOs, 1582427136 bytes 2024-09-17T23:16:52.262 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4100 IOs, 1620069376 bytes 2024-09-17T23:16:52.352 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4200 IOs, 1657387008 bytes 2024-09-17T23:16:52.459 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4300 IOs, 1698784256 bytes 2024-09-17T23:16:52.532 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4400 IOs, 1710801920 bytes 2024-09-17T23:16:52.624 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4500 IOs, 1741709312 bytes 2024-09-17T23:16:52.731 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4600 IOs, 1791254528 bytes 2024-09-17T23:16:52.818 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4700 IOs, 1812946944 bytes 2024-09-17T23:16:52.923 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4800 IOs, 1856450560 bytes 2024-09-17T23:16:53.044 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4900 IOs, 1926843392 bytes 2024-09-17T23:16:53.128 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5000 IOs, 1933949952 bytes 2024-09-17T23:16:53.243 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5100 IOs, 1987508224 bytes 2024-09-17T23:16:53.352 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5200 IOs, 2052046848 bytes 2024-09-17T23:16:53.441 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5300 IOs, 2069833728 bytes 2024-09-17T23:16:53.565 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5400 IOs, 2137392128 bytes 2024-09-17T23:16:53.673 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5500 IOs, 2185083904 bytes 2024-09-17T23:16:53.763 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5600 IOs, 2219077632 bytes 2024-09-17T23:16:53.846 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5700 IOs, 2233133056 bytes 2024-09-17T23:16:53.994 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5800 IOs, 2325981184 bytes 2024-09-17T23:16:54.067 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5900 IOs, 2337527808 bytes 2024-09-17T23:16:54.165 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ error_code=124 2024-09-17T23:16:54.165 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set -e 2024-09-17T23:16:54.165 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' 124 -eq 124 ']' 2024-09-17T23:16:54.165 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:16:54.165 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:16:54.166 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_5 2024-09-17T23:16:54.166 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_image_and_enable_mirror cluster2 mirror image_5 snapshot 128M 2024-09-17T23:16:54.166 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:16:54.166 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:16:54.166 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:16:54.166 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:16:54.166 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_5 2024-09-17T23:16:54.166 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:16:54.166 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=snapshot 2024-09-17T23:16:54.166 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n snapshot ']' 2024-09-17T23:16:54.167 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:16:54.167 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_image cluster2 mirror image_5 128M 2024-09-17T23:16:54.167 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:16:54.167 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:16:54.167 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:16:54.168 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:16:54.168 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_5 2024-09-17T23:16:54.168 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:16:54.168 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local size=128 2024-09-17T23:16:54.168 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n 128M ']' 2024-09-17T23:16:54.168 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ size=128M 2024-09-17T23:16:54.168 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:16:54.168 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_5 2024-09-17T23:16:54.217 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ is_pool_mirror_mode_image mirror 2024-09-17T23:16:54.217 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:16:54.217 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' image = image ']' 2024-09-17T23:16:54.217 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:16:54.217 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ enable_mirror cluster2 mirror image_5 snapshot 2024-09-17T23:16:54.217 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:16:54.217 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:16:54.217 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_5 2024-09-17T23:16:54.217 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=snapshot 2024-09-17T23:16:54.218 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_5 snapshot 2024-09-17T23:16:55.114 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Mirroring enabled 2024-09-17T23:16:55.124 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 info mirror/image_5 2024-09-17T23:16:55.165 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'image_5': 2024-09-17T23:16:55.165 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 128 MiB in 32 objects 2024-09-17T23:16:55.165 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:16:55.166 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 1 2024-09-17T23:16:55.166 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 12a55fd28237 2024-09-17T23:16:55.166 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.12a55fd28237 2024-09-17T23:16:55.166 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:16:55.166 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-17T23:16:55.166 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:16:55.166 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:16:55.166 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:16:54 2024 2024-09-17T23:16:55.166 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:16:54 2024 2024-09-17T23:16:55.166 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:16:54 2024 2024-09-17T23:16:55.166 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:16:55.166 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:16:55.166 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: bde8fdf5-e3d2-42d5-b63a-37026e062eaa 2024-09-17T23:16:55.167 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: true 2024-09-17T23:16:55.169 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n '' ']' 2024-09-17T23:16:55.169 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ stress_write_image cluster2 mirror image_5 2024-09-17T23:16:55.169 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:16:55.169 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:16:55.169 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_5 2024-09-17T23:16:55.169 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-17T23:16:55.171 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local duration=9 2024-09-17T23:16:55.171 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set +e 2024-09-17T23:16:55.171 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_5 --debug-rbd=20 --debug-journaler=20 2024-09-17T23:16:55.409 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:100 IOs, 111469568 bytes 2024-09-17T23:16:55.521 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:200 IOs, 136760320 bytes 2024-09-17T23:16:55.656 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:300 IOs, 214170624 bytes 2024-09-17T23:16:55.752 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:400 IOs, 249382912 bytes 2024-09-17T23:16:55.879 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:500 IOs, 314277888 bytes 2024-09-17T23:16:55.990 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:600 IOs, 358542336 bytes 2024-09-17T23:16:56.070 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:700 IOs, 372184064 bytes 2024-09-17T23:16:56.183 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:800 IOs, 422109184 bytes 2024-09-17T23:16:56.261 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:900 IOs, 440087552 bytes 2024-09-17T23:16:56.359 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1000 IOs, 475936768 bytes 2024-09-17T23:16:56.534 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1100 IOs, 593896448 bytes 2024-09-17T23:16:56.729 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1200 IOs, 739664896 bytes 2024-09-17T23:16:56.845 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1300 IOs, 790593536 bytes 2024-09-17T23:16:56.930 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1400 IOs, 806602752 bytes 2024-09-17T23:16:57.029 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1500 IOs, 841858048 bytes 2024-09-17T23:16:57.139 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1600 IOs, 878024704 bytes 2024-09-17T23:16:57.276 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1700 IOs, 953000960 bytes 2024-09-17T23:16:57.351 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1800 IOs, 957291520 bytes 2024-09-17T23:16:57.436 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1900 IOs, 975621120 bytes 2024-09-17T23:16:57.524 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2000 IOs, 996528128 bytes 2024-09-17T23:16:57.639 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2100 IOs, 1056629760 bytes 2024-09-17T23:16:57.763 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2200 IOs, 1102960640 bytes 2024-09-17T23:16:57.913 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2300 IOs, 1190006784 bytes 2024-09-17T23:16:57.984 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2400 IOs, 1196147712 bytes 2024-09-17T23:16:58.056 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2500 IOs, 1197086720 bytes 2024-09-17T23:16:58.202 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2600 IOs, 1278455808 bytes 2024-09-17T23:16:58.309 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2700 IOs, 1317026816 bytes 2024-09-17T23:16:58.416 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2800 IOs, 1354928128 bytes 2024-09-17T23:16:58.510 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2900 IOs, 1400298496 bytes 2024-09-17T23:16:58.623 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3000 IOs, 1434320896 bytes 2024-09-17T23:16:58.726 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3100 IOs, 1468645376 bytes 2024-09-17T23:16:58.804 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3200 IOs, 1487492096 bytes 2024-09-17T23:16:58.923 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3300 IOs, 1535352832 bytes 2024-09-17T23:16:58.995 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3400 IOs, 1538058240 bytes 2024-09-17T23:16:59.099 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3500 IOs, 1577704448 bytes 2024-09-17T23:16:59.185 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3600 IOs, 1601020928 bytes 2024-09-17T23:16:59.306 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3700 IOs, 1655282688 bytes 2024-09-17T23:16:59.406 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3800 IOs, 1689603072 bytes 2024-09-17T23:16:59.485 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3900 IOs, 1698718720 bytes 2024-09-17T23:16:59.637 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4000 IOs, 1788475392 bytes 2024-09-17T23:16:59.707 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4100 IOs, 1789084672 bytes 2024-09-17T23:16:59.838 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4200 IOs, 1855413248 bytes 2024-09-17T23:16:59.945 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4300 IOs, 1893107712 bytes 2024-09-17T23:17:00.024 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4400 IOs, 1906976768 bytes 2024-09-17T23:17:00.110 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4500 IOs, 1928132608 bytes 2024-09-17T23:17:00.217 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4600 IOs, 1967723520 bytes 2024-09-17T23:17:00.320 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4700 IOs, 2002132992 bytes 2024-09-17T23:17:00.394 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4800 IOs, 2007110656 bytes 2024-09-17T23:17:00.490 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4900 IOs, 2055868416 bytes 2024-09-17T23:17:00.595 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5000 IOs, 2072995840 bytes 2024-09-17T23:17:00.700 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5100 IOs, 2108351488 bytes 2024-09-17T23:17:00.812 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5200 IOs, 2153681920 bytes 2024-09-17T23:17:00.937 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5300 IOs, 2213539840 bytes 2024-09-17T23:17:01.068 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5400 IOs, 2274886656 bytes 2024-09-17T23:17:01.166 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5500 IOs, 2303952896 bytes 2024-09-17T23:17:01.310 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5600 IOs, 2388541440 bytes 2024-09-17T23:17:01.407 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5700 IOs, 2419943424 bytes 2024-09-17T23:17:01.509 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5800 IOs, 2456841216 bytes 2024-09-17T23:17:01.588 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5900 IOs, 2487500800 bytes 2024-09-17T23:17:01.728 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6000 IOs, 2572072960 bytes 2024-09-17T23:17:01.828 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6100 IOs, 2590902272 bytes 2024-09-17T23:17:01.953 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6200 IOs, 2648115200 bytes 2024-09-17T23:17:02.035 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6300 IOs, 2657340416 bytes 2024-09-17T23:17:02.158 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6400 IOs, 2712602624 bytes 2024-09-17T23:17:02.288 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6500 IOs, 2781227008 bytes 2024-09-17T23:17:02.490 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6600 IOs, 2918039552 bytes 2024-09-17T23:17:02.608 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6700 IOs, 2956729344 bytes 2024-09-17T23:17:02.740 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6800 IOs, 3023854592 bytes 2024-09-17T23:17:02.849 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6900 IOs, 3065620480 bytes 2024-09-17T23:17:02.983 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7000 IOs, 3143219200 bytes 2024-09-17T23:17:03.073 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7100 IOs, 3162739712 bytes 2024-09-17T23:17:03.171 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7200 IOs, 3187897344 bytes 2024-09-17T23:17:03.307 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7300 IOs, 3258572800 bytes 2024-09-17T23:17:03.439 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7400 IOs, 3319419904 bytes 2024-09-17T23:17:03.537 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7500 IOs, 3349216256 bytes 2024-09-17T23:17:03.624 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7600 IOs, 3367587840 bytes 2024-09-17T23:17:03.736 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7700 IOs, 3409875968 bytes 2024-09-17T23:17:03.821 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7800 IOs, 3427198976 bytes 2024-09-17T23:17:03.982 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7900 IOs, 3524747264 bytes 2024-09-17T23:17:04.101 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8000 IOs, 3567394816 bytes 2024-09-17T23:17:04.181 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ error_code=124 2024-09-17T23:17:04.181 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set -e 2024-09-17T23:17:04.181 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' 124 -eq 124 ']' 2024-09-17T23:17:04.181 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:17:04.181 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:17:04.182 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_6 2024-09-17T23:17:04.182 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_image_and_enable_mirror cluster2 mirror image_6 snapshot 128M 2024-09-17T23:17:04.182 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:17:04.182 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:17:04.182 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:17:04.184 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:17:04.184 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_6 2024-09-17T23:17:04.184 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:17:04.184 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=snapshot 2024-09-17T23:17:04.184 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n snapshot ']' 2024-09-17T23:17:04.184 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:17:04.184 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_image cluster2 mirror image_6 128M 2024-09-17T23:17:04.184 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:17:04.184 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:17:04.185 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:17:04.185 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:17:04.185 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_6 2024-09-17T23:17:04.185 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:17:04.185 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local size=128 2024-09-17T23:17:04.185 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n 128M ']' 2024-09-17T23:17:04.185 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ size=128M 2024-09-17T23:17:04.185 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:17:04.185 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_6 2024-09-17T23:17:04.234 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ is_pool_mirror_mode_image mirror 2024-09-17T23:17:04.234 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:17:04.234 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' image = image ']' 2024-09-17T23:17:04.234 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:17:04.234 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ enable_mirror cluster2 mirror image_6 snapshot 2024-09-17T23:17:04.234 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:17:04.234 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:17:04.235 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_6 2024-09-17T23:17:04.235 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=snapshot 2024-09-17T23:17:04.235 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_6 snapshot 2024-09-17T23:17:05.129 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Mirroring enabled 2024-09-17T23:17:05.140 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 info mirror/image_6 2024-09-17T23:17:05.180 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'image_6': 2024-09-17T23:17:05.181 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 128 MiB in 32 objects 2024-09-17T23:17:05.181 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:17:05.181 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 1 2024-09-17T23:17:05.181 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 12b15d8c10b7 2024-09-17T23:17:05.181 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.12b15d8c10b7 2024-09-17T23:17:05.181 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:17:05.181 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-17T23:17:05.181 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:17:05.181 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:17:05.181 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:17:04 2024 2024-09-17T23:17:05.181 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:17:04 2024 2024-09-17T23:17:05.181 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:17:04 2024 2024-09-17T23:17:05.182 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:17:05.182 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:17:05.182 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: d1b13a10-bcbb-4eb9-8b9a-30b9235cece2 2024-09-17T23:17:05.182 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: true 2024-09-17T23:17:05.184 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n '' ']' 2024-09-17T23:17:05.184 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ stress_write_image cluster2 mirror image_6 2024-09-17T23:17:05.185 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:17:05.185 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:17:05.185 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_6 2024-09-17T23:17:05.185 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-17T23:17:05.186 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local duration=11 2024-09-17T23:17:05.186 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set +e 2024-09-17T23:17:05.187 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_6 --debug-rbd=20 --debug-journaler=20 2024-09-17T23:17:05.397 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:100 IOs, 68058112 bytes 2024-09-17T23:17:05.488 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:200 IOs, 97233920 bytes 2024-09-17T23:17:05.588 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:300 IOs, 134196224 bytes 2024-09-17T23:17:05.686 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:400 IOs, 169832448 bytes 2024-09-17T23:17:05.785 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:500 IOs, 202574848 bytes 2024-09-17T23:17:05.915 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:600 IOs, 272722944 bytes 2024-09-17T23:17:05.998 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:700 IOs, 286199808 bytes 2024-09-17T23:17:06.144 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:800 IOs, 368772096 bytes 2024-09-17T23:17:06.252 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:900 IOs, 407221248 bytes 2024-09-17T23:17:06.351 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1000 IOs, 439490560 bytes 2024-09-17T23:17:06.420 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1100 IOs, 440090624 bytes 2024-09-17T23:17:06.528 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1200 IOs, 480629760 bytes 2024-09-17T23:17:06.611 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1300 IOs, 497956864 bytes 2024-09-17T23:17:06.680 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1400 IOs, 506526720 bytes 2024-09-17T23:17:06.807 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1500 IOs, 559744000 bytes 2024-09-17T23:17:06.950 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1600 IOs, 643093504 bytes 2024-09-17T23:17:07.057 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1700 IOs, 683405312 bytes 2024-09-17T23:17:07.172 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1800 IOs, 722140160 bytes 2024-09-17T23:17:07.291 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1900 IOs, 788879360 bytes 2024-09-17T23:17:07.400 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2000 IOs, 821765120 bytes 2024-09-17T23:17:07.498 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2100 IOs, 851824640 bytes 2024-09-17T23:17:07.590 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2200 IOs, 874177536 bytes 2024-09-17T23:17:07.680 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2300 IOs, 898515968 bytes 2024-09-17T23:17:07.765 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2400 IOs, 914522112 bytes 2024-09-17T23:17:07.953 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2500 IOs, 1045364736 bytes 2024-09-17T23:17:08.064 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2600 IOs, 1095393280 bytes 2024-09-17T23:17:08.186 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2700 IOs, 1142200320 bytes 2024-09-17T23:17:08.295 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2800 IOs, 1178221568 bytes 2024-09-17T23:17:08.398 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2900 IOs, 1209570304 bytes 2024-09-17T23:17:08.537 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3000 IOs, 1283580928 bytes 2024-09-17T23:17:08.683 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3100 IOs, 1371948032 bytes 2024-09-17T23:17:08.779 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3200 IOs, 1395612672 bytes 2024-09-17T23:17:08.885 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3300 IOs, 1451447296 bytes 2024-09-17T23:17:09.170 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3400 IOs, 1479347200 bytes 2024-09-17T23:17:09.256 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3500 IOs, 1500037120 bytes 2024-09-17T23:17:09.370 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3600 IOs, 1535014912 bytes 2024-09-17T23:17:09.518 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3700 IOs, 1618250752 bytes 2024-09-17T23:17:09.621 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3800 IOs, 1657347072 bytes 2024-09-17T23:17:09.713 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3900 IOs, 1681119232 bytes 2024-09-17T23:17:09.842 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4000 IOs, 1744225280 bytes 2024-09-17T23:17:09.962 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4100 IOs, 1796440064 bytes 2024-09-17T23:17:10.083 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4200 IOs, 1847760896 bytes 2024-09-17T23:17:10.168 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4300 IOs, 1864226816 bytes 2024-09-17T23:17:10.303 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4400 IOs, 1946688512 bytes 2024-09-17T23:17:10.439 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4500 IOs, 2008472576 bytes 2024-09-17T23:17:10.550 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4600 IOs, 2051376128 bytes 2024-09-17T23:17:10.654 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4700 IOs, 2090192896 bytes 2024-09-17T23:17:10.762 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4800 IOs, 2130743296 bytes 2024-09-17T23:17:10.867 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4900 IOs, 2168418304 bytes 2024-09-17T23:17:10.987 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5000 IOs, 2214136832 bytes 2024-09-17T23:17:11.065 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5100 IOs, 2222476288 bytes 2024-09-17T23:17:11.172 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5200 IOs, 2265031680 bytes 2024-09-17T23:17:11.287 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5300 IOs, 2303538176 bytes 2024-09-17T23:17:11.386 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5400 IOs, 2332020736 bytes 2024-09-17T23:17:11.485 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5500 IOs, 2362474496 bytes 2024-09-17T23:17:11.611 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5600 IOs, 2423468032 bytes 2024-09-17T23:17:11.701 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5700 IOs, 2450084864 bytes 2024-09-17T23:17:11.777 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5800 IOs, 2474356736 bytes 2024-09-17T23:17:11.907 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5900 IOs, 2516048896 bytes 2024-09-17T23:17:12.044 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6000 IOs, 2590407680 bytes 2024-09-17T23:17:12.164 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6100 IOs, 2641775616 bytes 2024-09-17T23:17:12.243 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6200 IOs, 2673020928 bytes 2024-09-17T23:17:12.337 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6300 IOs, 2687993856 bytes 2024-09-17T23:17:12.504 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6400 IOs, 2776652800 bytes 2024-09-17T23:17:12.629 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6500 IOs, 2828851200 bytes 2024-09-17T23:17:12.753 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6600 IOs, 2881367040 bytes 2024-09-17T23:17:12.845 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6700 IOs, 2902861824 bytes 2024-09-17T23:17:12.962 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6800 IOs, 2947545088 bytes 2024-09-17T23:17:13.066 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6900 IOs, 2983631872 bytes 2024-09-17T23:17:13.151 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7000 IOs, 2997165056 bytes 2024-09-17T23:17:13.265 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7100 IOs, 3037027328 bytes 2024-09-17T23:17:13.372 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7200 IOs, 3081004032 bytes 2024-09-17T23:17:13.472 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7300 IOs, 3103193088 bytes 2024-09-17T23:17:13.555 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7400 IOs, 3126298624 bytes 2024-09-17T23:17:13.663 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7500 IOs, 3151252480 bytes 2024-09-17T23:17:13.820 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7600 IOs, 3232585728 bytes 2024-09-17T23:17:13.951 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7700 IOs, 3292285952 bytes 2024-09-17T23:17:14.044 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7800 IOs, 3313581056 bytes 2024-09-17T23:17:14.119 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7900 IOs, 3314210816 bytes 2024-09-17T23:17:14.204 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8000 IOs, 3326879744 bytes 2024-09-17T23:17:14.304 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8100 IOs, 3360635904 bytes 2024-09-17T23:17:14.455 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8200 IOs, 3432630272 bytes 2024-09-17T23:17:14.533 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8300 IOs, 3447659520 bytes 2024-09-17T23:17:14.610 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8400 IOs, 3450450944 bytes 2024-09-17T23:17:14.717 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8500 IOs, 3487789056 bytes 2024-09-17T23:17:14.795 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8600 IOs, 3495048192 bytes 2024-09-17T23:17:14.919 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8700 IOs, 3546717184 bytes 2024-09-17T23:17:15.042 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8800 IOs, 3596069888 bytes 2024-09-17T23:17:15.159 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8900 IOs, 3646572544 bytes 2024-09-17T23:17:15.276 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9000 IOs, 3693003776 bytes 2024-09-17T23:17:15.349 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9100 IOs, 3693636608 bytes 2024-09-17T23:17:15.477 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9200 IOs, 3761580032 bytes 2024-09-17T23:17:15.608 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9300 IOs, 3792679936 bytes 2024-09-17T23:17:15.725 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9400 IOs, 3839721472 bytes 2024-09-17T23:17:15.812 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9500 IOs, 3855734784 bytes 2024-09-17T23:17:15.913 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9600 IOs, 3883339776 bytes 2024-09-17T23:17:16.014 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9700 IOs, 3913242624 bytes 2024-09-17T23:17:16.154 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9800 IOs, 3976516608 bytes 2024-09-17T23:17:16.192 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ error_code=124 2024-09-17T23:17:16.192 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set -e 2024-09-17T23:17:16.193 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' 124 -eq 124 ']' 2024-09-17T23:17:16.193 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:17:16.193 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:17:16.193 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_7 2024-09-17T23:17:16.193 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_image_and_enable_mirror cluster2 mirror image_7 snapshot 128M 2024-09-17T23:17:16.193 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:17:16.193 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:17:16.193 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:17:16.193 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:17:16.193 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_7 2024-09-17T23:17:16.193 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:17:16.193 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=snapshot 2024-09-17T23:17:16.193 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n snapshot ']' 2024-09-17T23:17:16.193 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:17:16.194 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_image cluster2 mirror image_7 128M 2024-09-17T23:17:16.194 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:17:16.194 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:17:16.194 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:17:16.194 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:17:16.194 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_7 2024-09-17T23:17:16.194 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:17:16.195 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local size=128 2024-09-17T23:17:16.195 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n 128M ']' 2024-09-17T23:17:16.195 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ size=128M 2024-09-17T23:17:16.195 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:17:16.195 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_7 2024-09-17T23:17:16.240 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ is_pool_mirror_mode_image mirror 2024-09-17T23:17:16.240 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:17:16.240 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' image = image ']' 2024-09-17T23:17:16.240 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:17:16.241 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ enable_mirror cluster2 mirror image_7 snapshot 2024-09-17T23:17:16.241 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:17:16.241 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:17:16.241 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_7 2024-09-17T23:17:16.241 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=snapshot 2024-09-17T23:17:16.241 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_7 snapshot 2024-09-17T23:17:17.146 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Mirroring enabled 2024-09-17T23:17:17.156 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 info mirror/image_7 2024-09-17T23:17:17.197 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'image_7': 2024-09-17T23:17:17.197 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 128 MiB in 32 objects 2024-09-17T23:17:17.197 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:17:17.197 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 1 2024-09-17T23:17:17.197 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 12bcb60346c8 2024-09-17T23:17:17.197 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.12bcb60346c8 2024-09-17T23:17:17.197 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:17:17.197 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-17T23:17:17.197 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:17:17.197 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:17:17.198 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:17:16 2024 2024-09-17T23:17:17.198 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:17:16 2024 2024-09-17T23:17:17.198 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:17:16 2024 2024-09-17T23:17:17.198 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:17:17.198 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:17:17.198 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: 3eb02a9d-e7e5-427f-9aed-20cdaea8a88e 2024-09-17T23:17:17.198 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: true 2024-09-17T23:17:17.200 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n '' ']' 2024-09-17T23:17:17.200 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ stress_write_image cluster2 mirror image_7 2024-09-17T23:17:17.200 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:17:17.200 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:17:17.201 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_7 2024-09-17T23:17:17.201 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-17T23:17:17.202 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local duration=7 2024-09-17T23:17:17.202 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set +e 2024-09-17T23:17:17.202 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_7 --debug-rbd=20 --debug-journaler=20 2024-09-17T23:17:17.401 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:100 IOs, 63307776 bytes 2024-09-17T23:17:17.498 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:200 IOs, 96039936 bytes 2024-09-17T23:17:17.571 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:300 IOs, 100888576 bytes 2024-09-17T23:17:17.640 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:400 IOs, 101504000 bytes 2024-09-17T23:17:17.752 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:500 IOs, 148911104 bytes 2024-09-17T23:17:17.842 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:600 IOs, 176259072 bytes 2024-09-17T23:17:17.927 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:700 IOs, 193945600 bytes 2024-09-17T23:17:18.048 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:800 IOs, 251251712 bytes 2024-09-17T23:17:18.144 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:900 IOs, 281448448 bytes 2024-09-17T23:17:18.296 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1000 IOs, 367589376 bytes 2024-09-17T23:17:18.429 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1100 IOs, 442249216 bytes 2024-09-17T23:17:18.500 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1200 IOs, 445760512 bytes 2024-09-17T23:17:18.574 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1300 IOs, 453338112 bytes 2024-09-17T23:17:18.673 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1400 IOs, 480879616 bytes 2024-09-17T23:17:18.766 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1500 IOs, 506604544 bytes 2024-09-17T23:17:18.854 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1600 IOs, 526485504 bytes 2024-09-17T23:17:18.949 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1700 IOs, 556264448 bytes 2024-09-17T23:17:19.039 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1800 IOs, 592559104 bytes 2024-09-17T23:17:19.193 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1900 IOs, 656923648 bytes 2024-09-17T23:17:19.308 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2000 IOs, 705986560 bytes 2024-09-17T23:17:19.400 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2100 IOs, 730144768 bytes 2024-09-17T23:17:19.528 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2200 IOs, 792303616 bytes 2024-09-17T23:17:19.598 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2300 IOs, 792907776 bytes 2024-09-17T23:17:19.700 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2400 IOs, 831116288 bytes 2024-09-17T23:17:19.822 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2500 IOs, 887171072 bytes 2024-09-17T23:17:19.945 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2600 IOs, 941366272 bytes 2024-09-17T23:17:20.030 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2700 IOs, 960903168 bytes 2024-09-17T23:17:20.175 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2800 IOs, 1041658880 bytes 2024-09-17T23:17:20.298 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2900 IOs, 1095270400 bytes 2024-09-17T23:17:20.377 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3000 IOs, 1106241536 bytes 2024-09-17T23:17:20.448 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3100 IOs, 1106853888 bytes 2024-09-17T23:17:20.600 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3200 IOs, 1141799936 bytes 2024-09-17T23:17:20.702 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3300 IOs, 1174943744 bytes 2024-09-17T23:17:20.793 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3400 IOs, 1194147840 bytes 2024-09-17T23:17:20.903 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3500 IOs, 1234403328 bytes 2024-09-17T23:17:21.017 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3600 IOs, 1279685632 bytes 2024-09-17T23:17:21.167 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3700 IOs, 1373134848 bytes 2024-09-17T23:17:21.267 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3800 IOs, 1401883648 bytes 2024-09-17T23:17:21.363 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3900 IOs, 1431005184 bytes 2024-09-17T23:17:21.529 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4000 IOs, 1527153664 bytes 2024-09-17T23:17:21.685 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4100 IOs, 1621742592 bytes 2024-09-17T23:17:21.862 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4200 IOs, 1732935680 bytes 2024-09-17T23:17:21.978 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4300 IOs, 1800680448 bytes 2024-09-17T23:17:22.092 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4400 IOs, 1824508928 bytes 2024-09-17T23:17:22.253 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4500 IOs, 1917736960 bytes 2024-09-17T23:17:22.349 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4600 IOs, 1944583168 bytes 2024-09-17T23:17:22.473 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4700 IOs, 2001721344 bytes 2024-09-17T23:17:22.565 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4800 IOs, 2027812864 bytes 2024-09-17T23:17:22.668 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4900 IOs, 2064188416 bytes 2024-09-17T23:17:22.792 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5000 IOs, 2125752320 bytes 2024-09-17T23:17:22.919 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5100 IOs, 2187487232 bytes 2024-09-17T23:17:23.076 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5200 IOs, 2281853952 bytes 2024-09-17T23:17:23.198 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5300 IOs, 2328750080 bytes 2024-09-17T23:17:23.287 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5400 IOs, 2347520000 bytes 2024-09-17T23:17:23.372 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5500 IOs, 2366309376 bytes 2024-09-17T23:17:23.476 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5600 IOs, 2413544448 bytes 2024-09-17T23:17:23.614 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5700 IOs, 2491731968 bytes 2024-09-17T23:17:23.713 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5800 IOs, 2511210496 bytes 2024-09-17T23:17:23.830 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5900 IOs, 2575749120 bytes 2024-09-17T23:17:23.951 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6000 IOs, 2612904960 bytes 2024-09-17T23:17:24.077 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6100 IOs, 2672182272 bytes 2024-09-17T23:17:24.171 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6200 IOs, 2692947968 bytes 2024-09-17T23:17:24.211 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ error_code=124 2024-09-17T23:17:24.211 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set -e 2024-09-17T23:17:24.211 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' 124 -eq 124 ']' 2024-09-17T23:17:24.211 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:17:24.212 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:17:24.212 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_8 2024-09-17T23:17:24.212 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_image_and_enable_mirror cluster2 mirror image_8 snapshot 128M 2024-09-17T23:17:24.212 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:17:24.212 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:17:24.212 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:17:24.212 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:17:24.212 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_8 2024-09-17T23:17:24.212 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:17:24.212 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=snapshot 2024-09-17T23:17:24.212 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n snapshot ']' 2024-09-17T23:17:24.212 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:17:24.212 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_image cluster2 mirror image_8 128M 2024-09-17T23:17:24.212 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:17:24.213 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:17:24.213 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:17:24.213 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:17:24.213 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_8 2024-09-17T23:17:24.213 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:17:24.213 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local size=128 2024-09-17T23:17:24.213 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n 128M ']' 2024-09-17T23:17:24.213 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ size=128M 2024-09-17T23:17:24.213 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:17:24.213 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_8 2024-09-17T23:17:24.260 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ is_pool_mirror_mode_image mirror 2024-09-17T23:17:24.260 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:17:24.261 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' image = image ']' 2024-09-17T23:17:24.261 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:17:24.261 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ enable_mirror cluster2 mirror image_8 snapshot 2024-09-17T23:17:24.261 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:17:24.261 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:17:24.261 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_8 2024-09-17T23:17:24.261 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=snapshot 2024-09-17T23:17:24.261 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_8 snapshot 2024-09-17T23:17:25.156 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Mirroring enabled 2024-09-17T23:17:25.165 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 info mirror/image_8 2024-09-17T23:17:25.206 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'image_8': 2024-09-17T23:17:25.206 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 128 MiB in 32 objects 2024-09-17T23:17:25.206 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:17:25.207 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 1 2024-09-17T23:17:25.207 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 12c818657bea 2024-09-17T23:17:25.207 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.12c818657bea 2024-09-17T23:17:25.207 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:17:25.207 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-17T23:17:25.207 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:17:25.207 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:17:25.207 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:17:24 2024 2024-09-17T23:17:25.207 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:17:24 2024 2024-09-17T23:17:25.207 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:17:24 2024 2024-09-17T23:17:25.207 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:17:25.207 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:17:25.208 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: ffca36ca-8ea2-4daa-91fd-451fc0c48309 2024-09-17T23:17:25.208 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: true 2024-09-17T23:17:25.210 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n '' ']' 2024-09-17T23:17:25.210 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ stress_write_image cluster2 mirror image_8 2024-09-17T23:17:25.210 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:17:25.210 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:17:25.210 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_8 2024-09-17T23:17:25.210 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-17T23:17:25.212 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local duration=7 2024-09-17T23:17:25.212 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set +e 2024-09-17T23:17:25.212 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_8 --debug-rbd=20 --debug-journaler=20 2024-09-17T23:17:25.385 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:100 IOs, 35900416 bytes 2024-09-17T23:17:25.458 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:200 IOs, 40740864 bytes 2024-09-17T23:17:25.538 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:300 IOs, 53287936 bytes 2024-09-17T23:17:25.606 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:400 IOs, 57805824 bytes 2024-09-17T23:17:25.737 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:500 IOs, 147993600 bytes 2024-09-17T23:17:25.923 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:600 IOs, 264671232 bytes 2024-09-17T23:17:26.036 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:700 IOs, 311880704 bytes 2024-09-17T23:17:26.174 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:800 IOs, 393039872 bytes 2024-09-17T23:17:26.293 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:900 IOs, 449039360 bytes 2024-09-17T23:17:26.442 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1000 IOs, 539128832 bytes 2024-09-17T23:17:26.561 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1100 IOs, 597700608 bytes 2024-09-17T23:17:26.666 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1200 IOs, 639652864 bytes 2024-09-17T23:17:26.747 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1300 IOs, 660097024 bytes 2024-09-17T23:17:26.846 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1400 IOs, 691373056 bytes 2024-09-17T23:17:26.919 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1500 IOs, 713491456 bytes 2024-09-17T23:17:27.032 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1600 IOs, 745184256 bytes 2024-09-17T23:17:27.170 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1700 IOs, 815059968 bytes 2024-09-17T23:17:27.253 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1800 IOs, 826329088 bytes 2024-09-17T23:17:27.394 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1900 IOs, 901723136 bytes 2024-09-17T23:17:27.494 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2000 IOs, 936007680 bytes 2024-09-17T23:17:27.596 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2100 IOs, 976717824 bytes 2024-09-17T23:17:27.746 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2200 IOs, 1071272960 bytes 2024-09-17T23:17:27.876 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2300 IOs, 1127392256 bytes 2024-09-17T23:17:27.993 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2400 IOs, 1180340224 bytes 2024-09-17T23:17:28.064 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2500 IOs, 1185437696 bytes 2024-09-17T23:17:28.162 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2600 IOs, 1231608832 bytes 2024-09-17T23:17:28.296 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2700 IOs, 1287527424 bytes 2024-09-17T23:17:28.440 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2800 IOs, 1380493312 bytes 2024-09-17T23:17:28.547 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2900 IOs, 1416624128 bytes 2024-09-17T23:17:28.640 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3000 IOs, 1444933632 bytes 2024-09-17T23:17:28.720 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3100 IOs, 1459336192 bytes 2024-09-17T23:17:28.817 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3200 IOs, 1490700288 bytes 2024-09-17T23:17:28.922 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3300 IOs, 1534004224 bytes 2024-09-17T23:17:29.033 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3400 IOs, 1573827584 bytes 2024-09-17T23:17:29.112 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3500 IOs, 1582594048 bytes 2024-09-17T23:17:29.226 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3600 IOs, 1635688448 bytes 2024-09-17T23:17:29.319 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3700 IOs, 1661775872 bytes 2024-09-17T23:17:29.404 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3800 IOs, 1679180800 bytes 2024-09-17T23:17:29.498 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3900 IOs, 1717734400 bytes 2024-09-17T23:17:29.586 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4000 IOs, 1726524416 bytes 2024-09-17T23:17:29.689 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4100 IOs, 1761476608 bytes 2024-09-17T23:17:29.800 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4200 IOs, 1805484032 bytes 2024-09-17T23:17:29.913 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4300 IOs, 1855768576 bytes 2024-09-17T23:17:30.007 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4400 IOs, 1881814016 bytes 2024-09-17T23:17:30.072 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4500 IOs, 1882408960 bytes 2024-09-17T23:17:30.185 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4600 IOs, 1930617856 bytes 2024-09-17T23:17:30.281 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4700 IOs, 1951323136 bytes 2024-09-17T23:17:30.378 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4800 IOs, 1984014336 bytes 2024-09-17T23:17:30.496 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4900 IOs, 2040442880 bytes 2024-09-17T23:17:30.613 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5000 IOs, 2093327360 bytes 2024-09-17T23:17:30.699 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5100 IOs, 2114327552 bytes 2024-09-17T23:17:30.783 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5200 IOs, 2126686208 bytes 2024-09-17T23:17:30.850 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5300 IOs, 2127315968 bytes 2024-09-17T23:17:31.011 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5400 IOs, 2217474048 bytes 2024-09-17T23:17:31.099 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5500 IOs, 2236067840 bytes 2024-09-17T23:17:31.181 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5600 IOs, 2250693632 bytes 2024-09-17T23:17:31.297 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5700 IOs, 2304442368 bytes 2024-09-17T23:17:31.389 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5800 IOs, 2320858112 bytes 2024-09-17T23:17:31.480 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5900 IOs, 2340598784 bytes 2024-09-17T23:17:31.586 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6000 IOs, 2378355712 bytes 2024-09-17T23:17:31.686 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6100 IOs, 2427513856 bytes 2024-09-17T23:17:31.810 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6200 IOs, 2471263232 bytes 2024-09-17T23:17:31.890 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6300 IOs, 2484655104 bytes 2024-09-17T23:17:31.960 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6400 IOs, 2488036352 bytes 2024-09-17T23:17:32.073 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6500 IOs, 2527981568 bytes 2024-09-17T23:17:32.192 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6600 IOs, 2579810304 bytes 2024-09-17T23:17:32.217 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ error_code=124 2024-09-17T23:17:32.217 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set -e 2024-09-17T23:17:32.217 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' 124 -eq 124 ']' 2024-09-17T23:17:32.218 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:17:32.218 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:17:32.218 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_9 2024-09-17T23:17:32.218 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_image_and_enable_mirror cluster2 mirror image_9 snapshot 128M 2024-09-17T23:17:32.218 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:17:32.218 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:17:32.218 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:17:32.218 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:17:32.218 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_9 2024-09-17T23:17:32.218 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:17:32.218 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=snapshot 2024-09-17T23:17:32.218 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n snapshot ']' 2024-09-17T23:17:32.218 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:17:32.218 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_image cluster2 mirror image_9 128M 2024-09-17T23:17:32.218 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:17:32.219 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:17:32.219 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:17:32.219 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:17:32.219 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_9 2024-09-17T23:17:32.219 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:17:32.219 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local size=128 2024-09-17T23:17:32.219 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n 128M ']' 2024-09-17T23:17:32.219 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ size=128M 2024-09-17T23:17:32.219 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:17:32.219 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_9 2024-09-17T23:17:32.265 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ is_pool_mirror_mode_image mirror 2024-09-17T23:17:32.266 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:17:32.266 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' image = image ']' 2024-09-17T23:17:32.266 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:17:32.266 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ enable_mirror cluster2 mirror image_9 snapshot 2024-09-17T23:17:32.266 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:17:32.266 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:17:32.266 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_9 2024-09-17T23:17:32.266 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=snapshot 2024-09-17T23:17:32.266 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_9 snapshot 2024-09-17T23:17:32.784 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Mirroring enabled 2024-09-17T23:17:32.794 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 info mirror/image_9 2024-09-17T23:17:32.832 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'image_9': 2024-09-17T23:17:32.832 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 128 MiB in 32 objects 2024-09-17T23:17:32.833 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:17:32.833 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 1 2024-09-17T23:17:32.833 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 12d4ec214fbf 2024-09-17T23:17:32.833 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.12d4ec214fbf 2024-09-17T23:17:32.833 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:17:32.833 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-17T23:17:32.833 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:17:32.833 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:17:32.833 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:17:32 2024 2024-09-17T23:17:32.833 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:17:32 2024 2024-09-17T23:17:32.833 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:17:32 2024 2024-09-17T23:17:32.833 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:17:32.833 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:17:32.833 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: bdde2c0b-4959-43dc-9438-c5f0c88af81b 2024-09-17T23:17:32.834 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: true 2024-09-17T23:17:32.836 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n '' ']' 2024-09-17T23:17:32.836 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ stress_write_image cluster2 mirror image_9 2024-09-17T23:17:32.836 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:17:32.836 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:17:32.836 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_9 2024-09-17T23:17:32.836 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-17T23:17:32.838 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local duration=14 2024-09-17T23:17:32.838 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set +e 2024-09-17T23:17:32.838 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_9 --debug-rbd=20 --debug-journaler=20 2024-09-17T23:17:33.046 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:100 IOs, 77894656 bytes 2024-09-17T23:17:33.157 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:200 IOs, 124333056 bytes 2024-09-17T23:17:33.265 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:300 IOs, 171835392 bytes 2024-09-17T23:17:33.380 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:400 IOs, 240459776 bytes 2024-09-17T23:17:33.485 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:500 IOs, 271740928 bytes 2024-09-17T23:17:33.594 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:600 IOs, 314258432 bytes 2024-09-17T23:17:33.710 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:700 IOs, 366557184 bytes 2024-09-17T23:17:33.778 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:800 IOs, 367176704 bytes 2024-09-17T23:17:33.906 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:900 IOs, 441561088 bytes 2024-09-17T23:17:34.045 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1000 IOs, 520309760 bytes 2024-09-17T23:17:34.163 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1100 IOs, 578605056 bytes 2024-09-17T23:17:34.270 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1200 IOs, 622005248 bytes 2024-09-17T23:17:34.383 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1300 IOs, 676178944 bytes 2024-09-17T23:17:34.494 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1400 IOs, 727136256 bytes 2024-09-17T23:17:34.597 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1500 IOs, 764691456 bytes 2024-09-17T23:17:34.676 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1600 IOs, 781546496 bytes 2024-09-17T23:17:34.759 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1700 IOs, 800133120 bytes 2024-09-17T23:17:34.846 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1800 IOs, 818248704 bytes 2024-09-17T23:17:34.995 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1900 IOs, 922875904 bytes 2024-09-17T23:17:35.126 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2000 IOs, 970927104 bytes 2024-09-17T23:17:35.255 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2100 IOs, 1030663168 bytes 2024-09-17T23:17:35.367 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2200 IOs, 1080189952 bytes 2024-09-17T23:17:35.497 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2300 IOs, 1152156672 bytes 2024-09-17T23:17:35.605 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2400 IOs, 1187554304 bytes 2024-09-17T23:17:35.683 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2500 IOs, 1205981184 bytes 2024-09-17T23:17:35.763 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2600 IOs, 1211626496 bytes 2024-09-17T23:17:35.865 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2700 IOs, 1248976896 bytes 2024-09-17T23:17:35.965 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2800 IOs, 1301144576 bytes 2024-09-17T23:17:36.104 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2900 IOs, 1358626816 bytes 2024-09-17T23:17:36.192 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3000 IOs, 1380887552 bytes 2024-09-17T23:17:36.279 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3100 IOs, 1404667904 bytes 2024-09-17T23:17:36.409 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3200 IOs, 1477310464 bytes 2024-09-17T23:17:36.521 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3300 IOs, 1522285568 bytes 2024-09-17T23:17:36.606 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3400 IOs, 1543285760 bytes 2024-09-17T23:17:36.711 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3500 IOs, 1586248704 bytes 2024-09-17T23:17:36.825 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3600 IOs, 1635481600 bytes 2024-09-17T23:17:36.941 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3700 IOs, 1695963136 bytes 2024-09-17T23:17:37.065 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3800 IOs, 1747473408 bytes 2024-09-17T23:17:37.208 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3900 IOs, 1829643264 bytes 2024-09-17T23:17:37.306 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4000 IOs, 1867506688 bytes 2024-09-17T23:17:37.446 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4100 IOs, 1945447424 bytes 2024-09-17T23:17:37.525 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4200 IOs, 1968715776 bytes 2024-09-17T23:17:37.606 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4300 IOs, 1973370880 bytes 2024-09-17T23:17:37.732 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4400 IOs, 2034995200 bytes 2024-09-17T23:17:37.868 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4500 IOs, 2106015744 bytes 2024-09-17T23:17:37.968 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4600 IOs, 2137324544 bytes 2024-09-17T23:17:38.087 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4700 IOs, 2197516288 bytes 2024-09-17T23:17:38.203 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4800 IOs, 2262121472 bytes 2024-09-17T23:17:38.301 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4900 IOs, 2276598784 bytes 2024-09-17T23:17:38.418 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5000 IOs, 2345550848 bytes 2024-09-17T23:17:38.525 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5100 IOs, 2381979648 bytes 2024-09-17T23:17:38.602 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5200 IOs, 2384555008 bytes 2024-09-17T23:17:38.666 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5300 IOs, 2385162240 bytes 2024-09-17T23:17:38.753 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5400 IOs, 2406558720 bytes 2024-09-17T23:17:38.849 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5500 IOs, 2434656256 bytes 2024-09-17T23:17:38.943 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5600 IOs, 2463995904 bytes 2024-09-17T23:17:39.020 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5700 IOs, 2472059904 bytes 2024-09-17T23:17:39.147 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5800 IOs, 2535490560 bytes 2024-09-17T23:17:39.270 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5900 IOs, 2605177856 bytes 2024-09-17T23:17:39.350 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6000 IOs, 2605798400 bytes 2024-09-17T23:17:39.445 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6100 IOs, 2632995840 bytes 2024-09-17T23:17:39.624 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6200 IOs, 2764313600 bytes 2024-09-17T23:17:39.731 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6300 IOs, 2797291520 bytes 2024-09-17T23:17:39.855 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6400 IOs, 2856474624 bytes 2024-09-17T23:17:39.969 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6500 IOs, 2905554944 bytes 2024-09-17T23:17:40.039 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6600 IOs, 2925285376 bytes 2024-09-17T23:17:40.149 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6700 IOs, 2951390208 bytes 2024-09-17T23:17:40.244 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6800 IOs, 2985629696 bytes 2024-09-17T23:17:40.352 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6900 IOs, 3017198592 bytes 2024-09-17T23:17:40.441 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7000 IOs, 3036794880 bytes 2024-09-17T23:17:40.544 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7100 IOs, 3073372160 bytes 2024-09-17T23:17:40.641 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7200 IOs, 3107281920 bytes 2024-09-17T23:17:40.738 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7300 IOs, 3135303680 bytes 2024-09-17T23:17:40.851 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7400 IOs, 3182330880 bytes 2024-09-17T23:17:40.962 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7500 IOs, 3221081088 bytes 2024-09-17T23:17:41.033 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7600 IOs, 3233637376 bytes 2024-09-17T23:17:41.197 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7700 IOs, 3319420928 bytes 2024-09-17T23:17:41.305 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7800 IOs, 3364139008 bytes 2024-09-17T23:17:41.413 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7900 IOs, 3398215680 bytes 2024-09-17T23:17:41.526 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8000 IOs, 3447294976 bytes 2024-09-17T23:17:41.623 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8100 IOs, 3470702592 bytes 2024-09-17T23:17:41.705 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8200 IOs, 3507429376 bytes 2024-09-17T23:17:41.835 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8300 IOs, 3561492480 bytes 2024-09-17T23:17:41.938 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8400 IOs, 3597677568 bytes 2024-09-17T23:17:42.007 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8500 IOs, 3598325760 bytes 2024-09-17T23:17:42.114 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8600 IOs, 3638071296 bytes 2024-09-17T23:17:42.204 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8700 IOs, 3664427008 bytes 2024-09-17T23:17:42.345 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8800 IOs, 3751898112 bytes 2024-09-17T23:17:42.444 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8900 IOs, 3774081024 bytes 2024-09-17T23:17:42.551 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9000 IOs, 3812125696 bytes 2024-09-17T23:17:42.645 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9100 IOs, 3839970304 bytes 2024-09-17T23:17:42.766 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9200 IOs, 3899090944 bytes 2024-09-17T23:17:42.856 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9300 IOs, 3918965760 bytes 2024-09-17T23:17:42.915 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-09-17T23:17:42.916 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-09-17T23:17:42.916 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2024-09-17T23:17:42.917 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi038.stderr:2024-09-17T23:17:42.915+0000 7fa57ed88640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 44265) UID: 1000 2024-09-17T23:17:42.951 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9400 IOs, 3935909888 bytes 2024-09-17T23:17:43.078 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9500 IOs, 3992479744 bytes 2024-09-17T23:17:43.228 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9600 IOs, 4074411008 bytes 2024-09-17T23:17:43.309 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9700 IOs, 4086609920 bytes 2024-09-17T23:17:43.382 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9800 IOs, 4101131264 bytes 2024-09-17T23:17:43.499 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9900 IOs, 4149894144 bytes 2024-09-17T23:17:43.594 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10000 IOs, 4172934144 bytes 2024-09-17T23:17:43.744 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10100 IOs, 4258961408 bytes 2024-09-17T23:17:43.851 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10200 IOs, 4294733824 bytes 2024-09-17T23:17:43.973 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10300 IOs, 4352244736 bytes 2024-09-17T23:17:44.063 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10400 IOs, 4370776064 bytes 2024-09-17T23:17:44.199 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10500 IOs, 4444395520 bytes 2024-09-17T23:17:44.290 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10600 IOs, 4466610176 bytes 2024-09-17T23:17:44.401 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10700 IOs, 4513603584 bytes 2024-09-17T23:17:44.485 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10800 IOs, 4530560000 bytes 2024-09-17T23:17:44.611 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10900 IOs, 4591084544 bytes 2024-09-17T23:17:44.707 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:11000 IOs, 4617439232 bytes 2024-09-17T23:17:44.885 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:11100 IOs, 4731156480 bytes 2024-09-17T23:17:45.007 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:11200 IOs, 4788285440 bytes 2024-09-17T23:17:45.105 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:11300 IOs, 4819877888 bytes 2024-09-17T23:17:45.202 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:11400 IOs, 4856227840 bytes 2024-09-17T23:17:45.303 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:11500 IOs, 4879541248 bytes 2024-09-17T23:17:45.420 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:11600 IOs, 4928821248 bytes 2024-09-17T23:17:45.510 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:11700 IOs, 4965061632 bytes 2024-09-17T23:17:45.631 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:11800 IOs, 5010829312 bytes 2024-09-17T23:17:45.725 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:11900 IOs, 5050573824 bytes 2024-09-17T23:17:45.837 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:12000 IOs, 5090055168 bytes 2024-09-17T23:17:45.986 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:12100 IOs, 5158774784 bytes 2024-09-17T23:17:46.066 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:12200 IOs, 5182759936 bytes 2024-09-17T23:17:46.214 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:12300 IOs, 5249488896 bytes 2024-09-17T23:17:46.289 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:12400 IOs, 5256653824 bytes 2024-09-17T23:17:46.431 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:12500 IOs, 5352770560 bytes 2024-09-17T23:17:46.529 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:12600 IOs, 5362231296 bytes 2024-09-17T23:17:46.630 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:12700 IOs, 5399242752 bytes 2024-09-17T23:17:46.842 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ error_code=124 2024-09-17T23:17:46.842 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set -e 2024-09-17T23:17:46.843 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' 124 -eq 124 ']' 2024-09-17T23:17:46.843 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:17:46.843 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:17:46.843 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_10 2024-09-17T23:17:46.843 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_image_and_enable_mirror cluster2 mirror image_10 snapshot 128M 2024-09-17T23:17:46.843 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:17:46.843 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:17:46.843 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:17:46.843 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:17:46.843 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_10 2024-09-17T23:17:46.843 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:17:46.843 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=snapshot 2024-09-17T23:17:46.843 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n snapshot ']' 2024-09-17T23:17:46.843 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:17:46.844 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_image cluster2 mirror image_10 128M 2024-09-17T23:17:46.844 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:17:46.844 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:17:46.844 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:17:46.844 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:17:46.845 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_10 2024-09-17T23:17:46.845 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:17:46.845 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local size=128 2024-09-17T23:17:46.845 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n 128M ']' 2024-09-17T23:17:46.845 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ size=128M 2024-09-17T23:17:46.845 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:17:46.845 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_10 2024-09-17T23:17:46.891 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ is_pool_mirror_mode_image mirror 2024-09-17T23:17:46.891 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:17:46.891 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' image = image ']' 2024-09-17T23:17:46.891 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:17:46.891 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ enable_mirror cluster2 mirror image_10 snapshot 2024-09-17T23:17:46.891 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:17:46.891 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:17:46.891 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_10 2024-09-17T23:17:46.891 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=snapshot 2024-09-17T23:17:46.892 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_10 snapshot 2024-09-17T23:17:47.806 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Mirroring enabled 2024-09-17T23:17:47.816 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 info mirror/image_10 2024-09-17T23:17:47.857 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'image_10': 2024-09-17T23:17:47.857 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 128 MiB in 32 objects 2024-09-17T23:17:47.857 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:17:47.857 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 1 2024-09-17T23:17:47.857 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 12e0e412d4c9 2024-09-17T23:17:47.857 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.12e0e412d4c9 2024-09-17T23:17:47.858 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:17:47.858 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-17T23:17:47.858 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:17:47.858 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:17:47.858 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:17:46 2024 2024-09-17T23:17:47.858 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:17:46 2024 2024-09-17T23:17:47.858 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:17:46 2024 2024-09-17T23:17:47.858 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:17:47.858 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:17:47.858 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: 2dc39d68-70b6-4566-beb0-8904894734d5 2024-09-17T23:17:47.858 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: true 2024-09-17T23:17:47.861 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n '' ']' 2024-09-17T23:17:47.861 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ stress_write_image cluster2 mirror image_10 2024-09-17T23:17:47.861 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:17:47.861 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:17:47.861 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_10 2024-09-17T23:17:47.861 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-17T23:17:47.863 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local duration=12 2024-09-17T23:17:47.863 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set +e 2024-09-17T23:17:47.863 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_10 --debug-rbd=20 --debug-journaler=20 2024-09-17T23:17:48.064 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:100 IOs, 58915840 bytes 2024-09-17T23:17:48.187 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:200 IOs, 105417728 bytes 2024-09-17T23:17:48.299 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:300 IOs, 151936000 bytes 2024-09-17T23:17:48.411 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:400 IOs, 206803968 bytes 2024-09-17T23:17:48.545 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:500 IOs, 266451968 bytes 2024-09-17T23:17:48.640 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:600 IOs, 292719616 bytes 2024-09-17T23:17:48.774 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:700 IOs, 375853056 bytes 2024-09-17T23:17:48.915 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:800 IOs, 434422784 bytes 2024-09-17T23:17:48.917 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-09-17T23:17:48.917 INFO:teuthology.orchestra.run:waiting for 600 2024-09-17T23:17:48.917 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-09-17T23:17:48.917 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-09-17T23:17:48.918 DEBUG:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-09-17T23:17:48.920 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-09-17T23:17:48.921 INFO:tasks.rbd_mirror_thrash:waiting for 91 secs before thrashing 2024-09-17T23:17:49.049 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:900 IOs, 482947072 bytes 2024-09-17T23:17:49.190 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1000 IOs, 555371520 bytes 2024-09-17T23:17:49.291 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1100 IOs, 577584128 bytes 2024-09-17T23:17:49.367 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1200 IOs, 589565952 bytes 2024-09-17T23:17:49.479 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1300 IOs, 606836736 bytes 2024-09-17T23:17:49.610 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1400 IOs, 667629568 bytes 2024-09-17T23:17:49.736 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1500 IOs, 720386048 bytes 2024-09-17T23:17:49.866 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1600 IOs, 781233152 bytes 2024-09-17T23:17:49.986 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1700 IOs, 820245504 bytes 2024-09-17T23:17:50.079 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1800 IOs, 843208704 bytes 2024-09-17T23:17:50.204 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1900 IOs, 899731456 bytes 2024-09-17T23:17:50.284 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2000 IOs, 907643904 bytes 2024-09-17T23:17:50.404 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2100 IOs, 955749376 bytes 2024-09-17T23:17:50.642 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2200 IOs, 1024450560 bytes 2024-09-17T23:17:50.718 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2300 IOs, 1025635328 bytes 2024-09-17T23:17:50.790 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2400 IOs, 1026253824 bytes 2024-09-17T23:17:50.892 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2500 IOs, 1062499328 bytes 2024-09-17T23:17:51.033 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2600 IOs, 1128762368 bytes 2024-09-17T23:17:51.156 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2700 IOs, 1179893760 bytes 2024-09-17T23:17:51.272 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2800 IOs, 1224391680 bytes 2024-09-17T23:17:51.366 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2900 IOs, 1249102848 bytes 2024-09-17T23:17:51.474 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3000 IOs, 1288613888 bytes 2024-09-17T23:17:51.606 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3100 IOs, 1364514816 bytes 2024-09-17T23:17:51.693 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3200 IOs, 1365104640 bytes 2024-09-17T23:17:51.792 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3300 IOs, 1394348032 bytes 2024-09-17T23:17:51.877 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3400 IOs, 1408412672 bytes 2024-09-17T23:17:52.001 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3500 IOs, 1461455872 bytes 2024-09-17T23:17:52.324 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3600 IOs, 1553423360 bytes 2024-09-17T23:17:52.434 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3700 IOs, 1591357440 bytes 2024-09-17T23:17:52.575 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3800 IOs, 1664087040 bytes 2024-09-17T23:17:52.694 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3900 IOs, 1708610560 bytes 2024-09-17T23:17:52.805 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4000 IOs, 1744756736 bytes 2024-09-17T23:17:52.892 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4100 IOs, 1760113664 bytes 2024-09-17T23:17:53.030 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4200 IOs, 1820550144 bytes 2024-09-17T23:17:53.151 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4300 IOs, 1876645888 bytes 2024-09-17T23:17:53.247 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4400 IOs, 1903073280 bytes 2024-09-17T23:17:53.342 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4500 IOs, 1926305792 bytes 2024-09-17T23:17:53.457 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4600 IOs, 1973716992 bytes 2024-09-17T23:17:53.602 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4700 IOs, 2049311744 bytes 2024-09-17T23:17:53.766 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4800 IOs, 2142049280 bytes 2024-09-17T23:17:53.852 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4900 IOs, 2151326720 bytes 2024-09-17T23:17:53.971 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5000 IOs, 2202324992 bytes 2024-09-17T23:17:54.079 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5100 IOs, 2244389888 bytes 2024-09-17T23:17:54.206 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5200 IOs, 2290863104 bytes 2024-09-17T23:17:54.294 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5300 IOs, 2305584128 bytes 2024-09-17T23:17:54.401 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5400 IOs, 2342761472 bytes 2024-09-17T23:17:54.533 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5500 IOs, 2412644352 bytes 2024-09-17T23:17:54.669 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5600 IOs, 2467651584 bytes 2024-09-17T23:17:54.797 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5700 IOs, 2519758848 bytes 2024-09-17T23:17:54.967 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5800 IOs, 2618016768 bytes 2024-09-17T23:17:55.059 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5900 IOs, 2638088192 bytes 2024-09-17T23:17:55.190 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6000 IOs, 2709696512 bytes 2024-09-17T23:17:55.369 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6100 IOs, 2794899456 bytes 2024-09-17T23:17:55.482 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6200 IOs, 2833361920 bytes 2024-09-17T23:17:55.646 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6300 IOs, 2880390144 bytes 2024-09-17T23:17:55.795 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6400 IOs, 2958457856 bytes 2024-09-17T23:17:55.932 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6500 IOs, 3032899584 bytes 2024-09-17T23:17:56.044 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6600 IOs, 3075970048 bytes 2024-09-17T23:17:56.175 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6700 IOs, 3130257408 bytes 2024-09-17T23:17:56.307 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6800 IOs, 3183179776 bytes 2024-09-17T23:17:56.428 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6900 IOs, 3239621632 bytes 2024-09-17T23:17:56.548 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7000 IOs, 3315718144 bytes 2024-09-17T23:17:56.704 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7100 IOs, 3395844096 bytes 2024-09-17T23:17:56.820 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7200 IOs, 3433329664 bytes 2024-09-17T23:17:56.897 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7300 IOs, 3434944512 bytes 2024-09-17T23:17:57.004 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7400 IOs, 3468449792 bytes 2024-09-17T23:17:57.141 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7500 IOs, 3533341696 bytes 2024-09-17T23:17:57.292 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7600 IOs, 3619100672 bytes 2024-09-17T23:17:57.436 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7700 IOs, 3690636288 bytes 2024-09-17T23:17:57.613 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7800 IOs, 3805361152 bytes 2024-09-17T23:17:57.719 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7900 IOs, 3841159168 bytes 2024-09-17T23:17:57.808 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8000 IOs, 3859832832 bytes 2024-09-17T23:17:57.944 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8100 IOs, 3926480896 bytes 2024-09-17T23:17:58.057 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8200 IOs, 3962620928 bytes 2024-09-17T23:17:58.156 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8300 IOs, 3987637248 bytes 2024-09-17T23:17:58.285 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8400 IOs, 4049244160 bytes 2024-09-17T23:17:58.364 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8500 IOs, 4049838080 bytes 2024-09-17T23:17:58.484 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8600 IOs, 4097422336 bytes 2024-09-17T23:17:58.586 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8700 IOs, 4137925632 bytes 2024-09-17T23:17:58.694 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8800 IOs, 4158358528 bytes 2024-09-17T23:17:58.830 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8900 IOs, 4222720000 bytes 2024-09-17T23:17:58.920 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9000 IOs, 4255951872 bytes 2024-09-17T23:17:59.069 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9100 IOs, 4313799680 bytes 2024-09-17T23:17:59.162 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9200 IOs, 4336396288 bytes 2024-09-17T23:17:59.238 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9300 IOs, 4339846144 bytes 2024-09-17T23:17:59.333 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9400 IOs, 4362344448 bytes 2024-09-17T23:17:59.430 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9500 IOs, 4387620864 bytes 2024-09-17T23:17:59.577 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9600 IOs, 4463134720 bytes 2024-09-17T23:17:59.658 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9700 IOs, 4474095616 bytes 2024-09-17T23:17:59.783 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9800 IOs, 4528945152 bytes 2024-09-17T23:17:59.869 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ error_code=124 2024-09-17T23:17:59.869 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set -e 2024-09-17T23:17:59.869 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' 124 -eq 124 ']' 2024-09-17T23:17:59.869 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:17:59.869 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:17:59.869 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_11 2024-09-17T23:17:59.869 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_image_and_enable_mirror cluster2 mirror image_11 snapshot 128M 2024-09-17T23:17:59.869 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:17:59.869 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:17:59.869 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:17:59.869 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:17:59.870 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_11 2024-09-17T23:17:59.870 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:17:59.870 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=snapshot 2024-09-17T23:17:59.870 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n snapshot ']' 2024-09-17T23:17:59.870 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:17:59.870 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_image cluster2 mirror image_11 128M 2024-09-17T23:17:59.870 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:17:59.870 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:17:59.870 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:17:59.870 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:17:59.870 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_11 2024-09-17T23:17:59.870 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:17:59.870 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local size=128 2024-09-17T23:17:59.871 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n 128M ']' 2024-09-17T23:17:59.871 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ size=128M 2024-09-17T23:17:59.871 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:17:59.871 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_11 2024-09-17T23:17:59.918 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ is_pool_mirror_mode_image mirror 2024-09-17T23:17:59.918 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:17:59.918 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' image = image ']' 2024-09-17T23:17:59.918 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:17:59.918 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ enable_mirror cluster2 mirror image_11 snapshot 2024-09-17T23:17:59.918 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:17:59.918 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:17:59.918 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_11 2024-09-17T23:17:59.918 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=snapshot 2024-09-17T23:17:59.918 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_11 snapshot 2024-09-17T23:18:00.836 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Mirroring enabled 2024-09-17T23:18:00.847 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 info mirror/image_11 2024-09-17T23:18:00.888 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'image_11': 2024-09-17T23:18:00.888 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 128 MiB in 32 objects 2024-09-17T23:18:00.888 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:18:00.888 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 1 2024-09-17T23:18:00.888 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 12f6f1246c4a 2024-09-17T23:18:00.888 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.12f6f1246c4a 2024-09-17T23:18:00.888 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:18:00.888 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-17T23:18:00.888 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:18:00.889 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:18:00.889 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:17:59 2024 2024-09-17T23:18:00.889 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:17:59 2024 2024-09-17T23:18:00.889 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:17:59 2024 2024-09-17T23:18:00.889 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:18:00.889 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:18:00.889 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: bd10d2f7-c8b2-4a32-94d9-b2af3d68bf2d 2024-09-17T23:18:00.889 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: true 2024-09-17T23:18:00.891 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n '' ']' 2024-09-17T23:18:00.891 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ stress_write_image cluster2 mirror image_11 2024-09-17T23:18:00.892 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:18:00.892 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:18:00.892 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_11 2024-09-17T23:18:00.892 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-17T23:18:00.893 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local duration=8 2024-09-17T23:18:00.894 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set +e 2024-09-17T23:18:00.894 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_11 --debug-rbd=20 --debug-journaler=20 2024-09-17T23:18:01.099 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:100 IOs, 58888192 bytes 2024-09-17T23:18:01.245 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:200 IOs, 131625984 bytes 2024-09-17T23:18:01.366 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:300 IOs, 184995840 bytes 2024-09-17T23:18:01.510 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:400 IOs, 260939776 bytes 2024-09-17T23:18:01.605 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:500 IOs, 296542208 bytes 2024-09-17T23:18:01.728 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:600 IOs, 343815168 bytes 2024-09-17T23:18:01.857 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:700 IOs, 408167424 bytes 2024-09-17T23:18:01.972 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:800 IOs, 445954048 bytes 2024-09-17T23:18:02.170 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:900 IOs, 592914432 bytes 2024-09-17T23:18:02.284 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1000 IOs, 632247296 bytes 2024-09-17T23:18:02.405 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1100 IOs, 691261440 bytes 2024-09-17T23:18:02.511 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1200 IOs, 724348928 bytes 2024-09-17T23:18:02.624 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1300 IOs, 766490624 bytes 2024-09-17T23:18:02.711 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1400 IOs, 782385152 bytes 2024-09-17T23:18:03.002 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1500 IOs, 835696640 bytes 2024-09-17T23:18:03.114 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1600 IOs, 879389696 bytes 2024-09-17T23:18:03.215 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1700 IOs, 907951104 bytes 2024-09-17T23:18:03.345 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1800 IOs, 970870784 bytes 2024-09-17T23:18:03.435 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1900 IOs, 989591552 bytes 2024-09-17T23:18:03.518 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2000 IOs, 996364288 bytes 2024-09-17T23:18:03.621 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2100 IOs, 1029912576 bytes 2024-09-17T23:18:03.745 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2200 IOs, 1087905792 bytes 2024-09-17T23:18:03.895 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2300 IOs, 1171838976 bytes 2024-09-17T23:18:04.019 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2400 IOs, 1228378112 bytes 2024-09-17T23:18:04.142 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2500 IOs, 1273652224 bytes 2024-09-17T23:18:04.284 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2600 IOs, 1356647424 bytes 2024-09-17T23:18:04.408 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2700 IOs, 1408109568 bytes 2024-09-17T23:18:04.519 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2800 IOs, 1449010176 bytes 2024-09-17T23:18:04.640 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2900 IOs, 1501950976 bytes 2024-09-17T23:18:04.726 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3000 IOs, 1515169792 bytes 2024-09-17T23:18:04.840 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3100 IOs, 1555332096 bytes 2024-09-17T23:18:04.952 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3200 IOs, 1599411200 bytes 2024-09-17T23:18:05.068 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3300 IOs, 1645512704 bytes 2024-09-17T23:18:05.198 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3400 IOs, 1705475072 bytes 2024-09-17T23:18:05.309 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3500 IOs, 1768642560 bytes 2024-09-17T23:18:05.470 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3600 IOs, 1840494592 bytes 2024-09-17T23:18:05.592 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3700 IOs, 1896276992 bytes 2024-09-17T23:18:05.728 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3800 IOs, 1962536960 bytes 2024-09-17T23:18:05.805 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3900 IOs, 1983473664 bytes 2024-09-17T23:18:05.915 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4000 IOs, 2011492352 bytes 2024-09-17T23:18:06.008 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4100 IOs, 2042374144 bytes 2024-09-17T23:18:06.084 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4200 IOs, 2042976256 bytes 2024-09-17T23:18:06.194 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4300 IOs, 2085267456 bytes 2024-09-17T23:18:06.294 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4400 IOs, 2111140864 bytes 2024-09-17T23:18:06.438 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4500 IOs, 2195292160 bytes 2024-09-17T23:18:06.574 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4600 IOs, 2257340416 bytes 2024-09-17T23:18:06.667 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4700 IOs, 2278358016 bytes 2024-09-17T23:18:06.783 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4800 IOs, 2323828736 bytes 2024-09-17T23:18:06.901 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4900 IOs, 2370899968 bytes 2024-09-17T23:18:07.001 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5000 IOs, 2401889280 bytes 2024-09-17T23:18:07.139 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5100 IOs, 2469914624 bytes 2024-09-17T23:18:07.237 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5200 IOs, 2496099328 bytes 2024-09-17T23:18:07.371 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5300 IOs, 2562608128 bytes 2024-09-17T23:18:07.445 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5400 IOs, 2563225600 bytes 2024-09-17T23:18:07.572 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5500 IOs, 2625637376 bytes 2024-09-17T23:18:07.711 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5600 IOs, 2688413696 bytes 2024-09-17T23:18:07.860 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5700 IOs, 2772698112 bytes 2024-09-17T23:18:07.969 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5800 IOs, 2804640768 bytes 2024-09-17T23:18:08.083 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5900 IOs, 2849540096 bytes 2024-09-17T23:18:08.219 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6000 IOs, 2920578048 bytes 2024-09-17T23:18:08.340 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6100 IOs, 2968049664 bytes 2024-09-17T23:18:08.424 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6200 IOs, 2986268672 bytes 2024-09-17T23:18:08.526 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6300 IOs, 3032581120 bytes 2024-09-17T23:18:08.636 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6400 IOs, 3054619648 bytes 2024-09-17T23:18:08.742 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6500 IOs, 3089122304 bytes 2024-09-17T23:18:08.853 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6600 IOs, 3127864320 bytes 2024-09-17T23:18:08.900 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ error_code=124 2024-09-17T23:18:08.900 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set -e 2024-09-17T23:18:08.900 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' 124 -eq 124 ']' 2024-09-17T23:18:08.901 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:18:08.901 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:18:08.901 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_12 2024-09-17T23:18:08.901 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_image_and_enable_mirror cluster2 mirror image_12 snapshot 128M 2024-09-17T23:18:08.901 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:18:08.901 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:18:08.901 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:18:08.901 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:18:08.901 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_12 2024-09-17T23:18:08.901 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:18:08.901 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=snapshot 2024-09-17T23:18:08.901 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n snapshot ']' 2024-09-17T23:18:08.901 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:18:08.901 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_image cluster2 mirror image_12 128M 2024-09-17T23:18:08.902 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:18:08.902 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:18:08.902 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:18:08.902 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:18:08.902 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_12 2024-09-17T23:18:08.902 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:18:08.902 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local size=128 2024-09-17T23:18:08.902 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n 128M ']' 2024-09-17T23:18:08.902 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ size=128M 2024-09-17T23:18:08.902 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:18:08.902 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_12 2024-09-17T23:18:08.952 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ is_pool_mirror_mode_image mirror 2024-09-17T23:18:08.952 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:18:08.952 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' image = image ']' 2024-09-17T23:18:08.952 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:18:08.952 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ enable_mirror cluster2 mirror image_12 snapshot 2024-09-17T23:18:08.952 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:18:08.952 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:18:08.952 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_12 2024-09-17T23:18:08.952 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=snapshot 2024-09-17T23:18:08.952 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_12 snapshot 2024-09-17T23:18:09.853 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Mirroring enabled 2024-09-17T23:18:09.864 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 info mirror/image_12 2024-09-17T23:18:09.904 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'image_12': 2024-09-17T23:18:09.904 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 128 MiB in 32 objects 2024-09-17T23:18:09.904 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:18:09.904 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 1 2024-09-17T23:18:09.904 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 1302e6b7c78c 2024-09-17T23:18:09.905 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.1302e6b7c78c 2024-09-17T23:18:09.905 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:18:09.905 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-17T23:18:09.905 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:18:09.905 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:18:09.905 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:18:08 2024 2024-09-17T23:18:09.905 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:18:08 2024 2024-09-17T23:18:09.905 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:18:08 2024 2024-09-17T23:18:09.905 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:18:09.905 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:18:09.905 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: 1e208d5a-c3ae-4b3c-bb32-593db1dbda33 2024-09-17T23:18:09.905 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: true 2024-09-17T23:18:09.907 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n '' ']' 2024-09-17T23:18:09.908 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ stress_write_image cluster2 mirror image_12 2024-09-17T23:18:09.908 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:18:09.908 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:18:09.908 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_12 2024-09-17T23:18:09.908 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-17T23:18:09.909 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local duration=7 2024-09-17T23:18:09.910 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set +e 2024-09-17T23:18:09.910 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_12 --debug-rbd=20 --debug-journaler=20 2024-09-17T23:18:10.102 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:100 IOs, 35988480 bytes 2024-09-17T23:18:10.171 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:200 IOs, 36589568 bytes 2024-09-17T23:18:10.333 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:300 IOs, 145148928 bytes 2024-09-17T23:18:10.416 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:400 IOs, 157862912 bytes 2024-09-17T23:18:10.562 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:500 IOs, 232423424 bytes 2024-09-17T23:18:10.646 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:600 IOs, 250798080 bytes 2024-09-17T23:18:10.718 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:700 IOs, 257021952 bytes 2024-09-17T23:18:10.840 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:800 IOs, 329356288 bytes 2024-09-17T23:18:10.940 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:900 IOs, 349582336 bytes 2024-09-17T23:18:11.019 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1000 IOs, 354990080 bytes 2024-09-17T23:18:11.119 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1100 IOs, 392345600 bytes 2024-09-17T23:18:11.260 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1200 IOs, 461810688 bytes 2024-09-17T23:18:11.365 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1300 IOs, 502618112 bytes 2024-09-17T23:18:11.481 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1400 IOs, 548127744 bytes 2024-09-17T23:18:11.568 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1500 IOs, 568920064 bytes 2024-09-17T23:18:11.682 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1600 IOs, 612401152 bytes 2024-09-17T23:18:11.813 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1700 IOs, 691915776 bytes 2024-09-17T23:18:11.910 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1800 IOs, 703426560 bytes 2024-09-17T23:18:11.949 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi038.stderr:2024-09-17T23:18:11.947+0000 7f63819c3640 -1 rbd::mirror::ImageReplayer: 0x560eefff6500 [3/1e208d5a-c3ae-4b3c-bb32-593db1dbda33] start: already running 2024-09-17T23:18:12.041 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1900 IOs, 781404160 bytes 2024-09-17T23:18:12.145 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2000 IOs, 790257664 bytes 2024-09-17T23:18:12.273 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2100 IOs, 851192832 bytes 2024-09-17T23:18:12.384 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2200 IOs, 899777536 bytes 2024-09-17T23:18:12.483 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2300 IOs, 929571840 bytes 2024-09-17T23:18:12.586 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2400 IOs, 964453376 bytes 2024-09-17T23:18:12.705 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2500 IOs, 1010693120 bytes 2024-09-17T23:18:12.805 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2600 IOs, 1042727936 bytes 2024-09-17T23:18:12.891 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2700 IOs, 1059034112 bytes 2024-09-17T23:18:13.044 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2800 IOs, 1142274048 bytes 2024-09-17T23:18:13.155 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2900 IOs, 1181598720 bytes 2024-09-17T23:18:13.333 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3000 IOs, 1292482560 bytes 2024-09-17T23:18:13.458 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3100 IOs, 1353705472 bytes 2024-09-17T23:18:13.563 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3200 IOs, 1399648256 bytes 2024-09-17T23:18:13.660 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3300 IOs, 1428154368 bytes 2024-09-17T23:18:13.766 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3400 IOs, 1467641856 bytes 2024-09-17T23:18:13.843 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3500 IOs, 1472742400 bytes 2024-09-17T23:18:13.956 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3600 IOs, 1525513216 bytes 2024-09-17T23:18:14.089 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3700 IOs, 1586459648 bytes 2024-09-17T23:18:14.187 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3800 IOs, 1632199680 bytes 2024-09-17T23:18:14.295 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3900 IOs, 1669857280 bytes 2024-09-17T23:18:14.432 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4000 IOs, 1723520000 bytes 2024-09-17T23:18:14.533 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4100 IOs, 1759978496 bytes 2024-09-17T23:18:14.650 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4200 IOs, 1804884992 bytes 2024-09-17T23:18:14.769 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4300 IOs, 1855153152 bytes 2024-09-17T23:18:14.900 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4400 IOs, 1925352448 bytes 2024-09-17T23:18:14.997 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4500 IOs, 1963957248 bytes 2024-09-17T23:18:15.102 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4600 IOs, 1993603072 bytes 2024-09-17T23:18:15.208 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4700 IOs, 2032661504 bytes 2024-09-17T23:18:15.353 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4800 IOs, 2110150656 bytes 2024-09-17T23:18:15.465 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4900 IOs, 2154409984 bytes 2024-09-17T23:18:15.571 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5000 IOs, 2202568704 bytes 2024-09-17T23:18:15.689 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5100 IOs, 2243349504 bytes 2024-09-17T23:18:15.784 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5200 IOs, 2265160704 bytes 2024-09-17T23:18:15.889 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5300 IOs, 2299352064 bytes 2024-09-17T23:18:15.965 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5400 IOs, 2300574720 bytes 2024-09-17T23:18:16.111 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5500 IOs, 2400506880 bytes 2024-09-17T23:18:16.223 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5600 IOs, 2425375744 bytes 2024-09-17T23:18:16.360 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5700 IOs, 2493288448 bytes 2024-09-17T23:18:16.454 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5800 IOs, 2533956608 bytes 2024-09-17T23:18:16.559 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5900 IOs, 2552770560 bytes 2024-09-17T23:18:16.694 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6000 IOs, 2629126144 bytes 2024-09-17T23:18:16.775 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6100 IOs, 2629737472 bytes 2024-09-17T23:18:16.815 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.3246801950875533) > weight (0.25) 2024-09-17T23:18:16.815 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.4473825205230082) > weight (0.25) 2024-09-17T23:18:16.815 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.8140047385749462) > weight (0.25) 2024-09-17T23:18:16.815 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.6380133912526291) > weight (0.25) 2024-09-17T23:18:16.816 INFO:tasks.rbd_mirror_thrash:waiting for 86 secs before thrashing 2024-09-17T23:18:16.850 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6200 IOs, 2631261184 bytes 2024-09-17T23:18:16.915 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ error_code=124 2024-09-17T23:18:16.915 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set -e 2024-09-17T23:18:16.915 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' 124 -eq 124 ']' 2024-09-17T23:18:16.915 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:18:16.915 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:18:16.915 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_13 2024-09-17T23:18:16.915 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_image_and_enable_mirror cluster2 mirror image_13 snapshot 128M 2024-09-17T23:18:16.915 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:18:16.915 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:18:16.915 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:18:16.916 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:18:16.916 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_13 2024-09-17T23:18:16.916 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:18:16.916 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=snapshot 2024-09-17T23:18:16.916 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n snapshot ']' 2024-09-17T23:18:16.916 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:18:16.916 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_image cluster2 mirror image_13 128M 2024-09-17T23:18:16.916 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:18:16.916 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:18:16.916 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:18:16.916 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:18:16.916 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_13 2024-09-17T23:18:16.916 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:18:16.916 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local size=128 2024-09-17T23:18:16.917 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n 128M ']' 2024-09-17T23:18:16.917 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ size=128M 2024-09-17T23:18:16.917 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:18:16.917 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_13 2024-09-17T23:18:16.964 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ is_pool_mirror_mode_image mirror 2024-09-17T23:18:16.964 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:18:16.964 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' image = image ']' 2024-09-17T23:18:16.964 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:18:16.964 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ enable_mirror cluster2 mirror image_13 snapshot 2024-09-17T23:18:16.964 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:18:16.964 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:18:16.964 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_13 2024-09-17T23:18:16.964 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=snapshot 2024-09-17T23:18:16.965 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_13 snapshot 2024-09-17T23:18:17.855 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Mirroring enabled 2024-09-17T23:18:17.864 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 info mirror/image_13 2024-09-17T23:18:17.902 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'image_13': 2024-09-17T23:18:17.903 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 128 MiB in 32 objects 2024-09-17T23:18:17.903 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:18:17.903 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 1 2024-09-17T23:18:17.903 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 130e38aa0581 2024-09-17T23:18:17.903 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.130e38aa0581 2024-09-17T23:18:17.903 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:18:17.903 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-17T23:18:17.903 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:18:17.903 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:18:17.903 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:18:16 2024 2024-09-17T23:18:17.903 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:18:16 2024 2024-09-17T23:18:17.903 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:18:16 2024 2024-09-17T23:18:17.903 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:18:17.904 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:18:17.904 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: 9ebe7c6f-3aa1-48f9-82b3-c20461625f84 2024-09-17T23:18:17.904 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: true 2024-09-17T23:18:17.906 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n '' ']' 2024-09-17T23:18:17.906 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ stress_write_image cluster2 mirror image_13 2024-09-17T23:18:17.906 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:18:17.906 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:18:17.906 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_13 2024-09-17T23:18:17.906 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-17T23:18:17.908 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local duration=11 2024-09-17T23:18:17.908 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set +e 2024-09-17T23:18:17.908 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_13 --debug-rbd=20 --debug-journaler=20 2024-09-17T23:18:18.085 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:100 IOs, 28991488 bytes 2024-09-17T23:18:18.196 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:200 IOs, 78520320 bytes 2024-09-17T23:18:18.508 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:300 IOs, 119665664 bytes 2024-09-17T23:18:18.607 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:400 IOs, 161810432 bytes 2024-09-17T23:18:18.759 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:500 IOs, 242516992 bytes 2024-09-17T23:18:18.842 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:600 IOs, 256056320 bytes 2024-09-17T23:18:18.930 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:700 IOs, 273593344 bytes 2024-09-17T23:18:19.034 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:800 IOs, 309343232 bytes 2024-09-17T23:18:19.117 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:900 IOs, 317405184 bytes 2024-09-17T23:18:19.217 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1000 IOs, 342562816 bytes 2024-09-17T23:18:19.288 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1100 IOs, 343207936 bytes 2024-09-17T23:18:19.456 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1200 IOs, 451340288 bytes 2024-09-17T23:18:19.579 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1300 IOs, 500847616 bytes 2024-09-17T23:18:19.668 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1400 IOs, 521494528 bytes 2024-09-17T23:18:19.767 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1500 IOs, 571719680 bytes 2024-09-17T23:18:19.938 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1600 IOs, 645817344 bytes 2024-09-17T23:18:20.037 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1700 IOs, 670899200 bytes 2024-09-17T23:18:20.167 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1800 IOs, 734981120 bytes 2024-09-17T23:18:20.275 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1900 IOs, 771587072 bytes 2024-09-17T23:18:20.368 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2000 IOs, 794810368 bytes 2024-09-17T23:18:20.466 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2100 IOs, 822609920 bytes 2024-09-17T23:18:20.663 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2200 IOs, 952627200 bytes 2024-09-17T23:18:20.796 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2300 IOs, 1015627776 bytes 2024-09-17T23:18:20.883 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2400 IOs, 1035888640 bytes 2024-09-17T23:18:20.976 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2500 IOs, 1060887552 bytes 2024-09-17T23:18:21.111 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2600 IOs, 1131771904 bytes 2024-09-17T23:18:21.235 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2700 IOs, 1209208832 bytes 2024-09-17T23:18:21.382 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2800 IOs, 1277259776 bytes 2024-09-17T23:18:21.511 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2900 IOs, 1335718912 bytes 2024-09-17T23:18:21.583 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3000 IOs, 1338024960 bytes 2024-09-17T23:18:21.659 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3100 IOs, 1341955072 bytes 2024-09-17T23:18:21.756 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3200 IOs, 1366249472 bytes 2024-09-17T23:18:21.841 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3300 IOs, 1381135360 bytes 2024-09-17T23:18:21.922 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3400 IOs, 1389440000 bytes 2024-09-17T23:18:22.117 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3500 IOs, 1523359744 bytes 2024-09-17T23:18:22.237 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3600 IOs, 1571800064 bytes 2024-09-17T23:18:22.340 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3700 IOs, 1608350720 bytes 2024-09-17T23:18:22.489 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3800 IOs, 1687600128 bytes 2024-09-17T23:18:22.587 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3900 IOs, 1722698752 bytes 2024-09-17T23:18:22.670 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4000 IOs, 1723309056 bytes 2024-09-17T23:18:22.803 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4100 IOs, 1783418880 bytes 2024-09-17T23:18:22.909 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4200 IOs, 1821401088 bytes 2024-09-17T23:18:23.012 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4300 IOs, 1851036672 bytes 2024-09-17T23:18:23.103 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4400 IOs, 1871411200 bytes 2024-09-17T23:18:23.213 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4500 IOs, 1906332672 bytes 2024-09-17T23:18:23.351 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4600 IOs, 1977212928 bytes 2024-09-17T23:18:23.428 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4700 IOs, 1992620032 bytes 2024-09-17T23:18:23.538 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4800 IOs, 2024458240 bytes 2024-09-17T23:18:23.617 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4900 IOs, 2028101632 bytes 2024-09-17T23:18:23.709 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5000 IOs, 2050169856 bytes 2024-09-17T23:18:23.800 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5100 IOs, 2067023872 bytes 2024-09-17T23:18:23.899 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5200 IOs, 2091891712 bytes 2024-09-17T23:18:24.008 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5300 IOs, 2135574528 bytes 2024-09-17T23:18:24.141 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5400 IOs, 2192589824 bytes 2024-09-17T23:18:24.256 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5500 IOs, 2236455936 bytes 2024-09-17T23:18:24.377 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5600 IOs, 2316092416 bytes 2024-09-17T23:18:24.508 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5700 IOs, 2342365184 bytes 2024-09-17T23:18:24.623 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5800 IOs, 2390602752 bytes 2024-09-17T23:18:24.726 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5900 IOs, 2425025536 bytes 2024-09-17T23:18:24.848 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6000 IOs, 2468045824 bytes 2024-09-17T23:18:24.987 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6100 IOs, 2532973568 bytes 2024-09-17T23:18:25.058 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6200 IOs, 2533582848 bytes 2024-09-17T23:18:25.176 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6300 IOs, 2585260032 bytes 2024-09-17T23:18:25.281 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6400 IOs, 2611625984 bytes 2024-09-17T23:18:25.376 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6500 IOs, 2650443776 bytes 2024-09-17T23:18:25.472 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6600 IOs, 2658260992 bytes 2024-09-17T23:18:25.568 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6700 IOs, 2678340608 bytes 2024-09-17T23:18:25.661 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6800 IOs, 2697197568 bytes 2024-09-17T23:18:25.828 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6900 IOs, 2794535936 bytes 2024-09-17T23:18:25.920 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7000 IOs, 2815687680 bytes 2024-09-17T23:18:26.027 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7100 IOs, 2853115904 bytes 2024-09-17T23:18:26.145 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7200 IOs, 2897389568 bytes 2024-09-17T23:18:26.250 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7300 IOs, 2938352640 bytes 2024-09-17T23:18:26.408 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7400 IOs, 3016544256 bytes 2024-09-17T23:18:26.526 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7500 IOs, 3062606848 bytes 2024-09-17T23:18:26.617 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7600 IOs, 3080265728 bytes 2024-09-17T23:18:26.701 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7700 IOs, 3089524736 bytes 2024-09-17T23:18:26.802 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7800 IOs, 3118707712 bytes 2024-09-17T23:18:26.928 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7900 IOs, 3161531392 bytes 2024-09-17T23:18:27.022 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8000 IOs, 3182052352 bytes 2024-09-17T23:18:27.143 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8100 IOs, 3230481408 bytes 2024-09-17T23:18:27.241 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8200 IOs, 3268357120 bytes 2024-09-17T23:18:27.417 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8300 IOs, 3359963136 bytes 2024-09-17T23:18:27.560 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8400 IOs, 3430986752 bytes 2024-09-17T23:18:27.660 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8500 IOs, 3461468160 bytes 2024-09-17T23:18:27.768 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8600 IOs, 3496969216 bytes 2024-09-17T23:18:27.878 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8700 IOs, 3530559488 bytes 2024-09-17T23:18:27.962 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8800 IOs, 3540678656 bytes 2024-09-17T23:18:28.082 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8900 IOs, 3590139904 bytes 2024-09-17T23:18:28.211 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9000 IOs, 3650553856 bytes 2024-09-17T23:18:28.331 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9100 IOs, 3704544256 bytes 2024-09-17T23:18:28.441 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9200 IOs, 3747139584 bytes 2024-09-17T23:18:28.551 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9300 IOs, 3785890816 bytes 2024-09-17T23:18:28.646 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9400 IOs, 3805708288 bytes 2024-09-17T23:18:28.772 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9500 IOs, 3863192576 bytes 2024-09-17T23:18:28.896 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9600 IOs, 3911021568 bytes 2024-09-17T23:18:28.898 INFO:tasks.ceph.cluster2.osd.2.smithi038.stderr:2024-09-17T23:18:28.895+0000 7f1c4b04a640 -1 reset not still connected to 0x561cdff24680 2024-09-17T23:18:28.915 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ error_code=124 2024-09-17T23:18:28.915 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set -e 2024-09-17T23:18:28.915 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' 124 -eq 124 ']' 2024-09-17T23:18:28.915 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:18:28.916 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:18:28.916 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_14 2024-09-17T23:18:28.916 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_image_and_enable_mirror cluster2 mirror image_14 snapshot 128M 2024-09-17T23:18:28.916 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:18:28.916 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:18:28.916 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:18:28.916 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:18:28.916 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_14 2024-09-17T23:18:28.916 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:18:28.916 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=snapshot 2024-09-17T23:18:28.917 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n snapshot ']' 2024-09-17T23:18:28.917 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:18:28.917 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_image cluster2 mirror image_14 128M 2024-09-17T23:18:28.917 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:18:28.917 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:18:28.917 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:18:28.917 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:18:28.917 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_14 2024-09-17T23:18:28.917 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:18:28.917 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local size=128 2024-09-17T23:18:28.917 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n 128M ']' 2024-09-17T23:18:28.917 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ size=128M 2024-09-17T23:18:28.918 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:18:28.918 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_14 2024-09-17T23:18:28.966 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ is_pool_mirror_mode_image mirror 2024-09-17T23:18:28.966 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:18:28.966 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' image = image ']' 2024-09-17T23:18:28.966 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:18:28.967 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ enable_mirror cluster2 mirror image_14 snapshot 2024-09-17T23:18:28.967 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:18:28.967 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:18:28.967 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_14 2024-09-17T23:18:28.967 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=snapshot 2024-09-17T23:18:28.967 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_14 snapshot 2024-09-17T23:18:29.888 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Mirroring enabled 2024-09-17T23:18:29.898 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 info mirror/image_14 2024-09-17T23:18:29.938 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'image_14': 2024-09-17T23:18:29.939 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 128 MiB in 32 objects 2024-09-17T23:18:29.939 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:18:29.939 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 1 2024-09-17T23:18:29.939 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 131a1619ba67 2024-09-17T23:18:29.939 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.131a1619ba67 2024-09-17T23:18:29.939 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:18:29.939 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-17T23:18:29.939 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:18:29.939 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:18:29.940 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:18:28 2024 2024-09-17T23:18:29.940 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:18:28 2024 2024-09-17T23:18:29.940 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:18:28 2024 2024-09-17T23:18:29.940 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:18:29.940 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:18:29.940 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: 115aba0c-a419-49de-b089-0afc43656b9c 2024-09-17T23:18:29.940 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: true 2024-09-17T23:18:29.942 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n '' ']' 2024-09-17T23:18:29.942 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ stress_write_image cluster2 mirror image_14 2024-09-17T23:18:29.943 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:18:29.943 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:18:29.943 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_14 2024-09-17T23:18:29.943 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-17T23:18:29.944 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local duration=10 2024-09-17T23:18:29.944 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set +e 2024-09-17T23:18:29.944 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_14 --debug-rbd=20 --debug-journaler=20 2024-09-17T23:18:30.155 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:100 IOs, 59612160 bytes 2024-09-17T23:18:30.251 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:200 IOs, 86676480 bytes 2024-09-17T23:18:30.349 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:300 IOs, 109244416 bytes 2024-09-17T23:18:30.430 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:400 IOs, 123408384 bytes 2024-09-17T23:18:30.524 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:500 IOs, 140335104 bytes 2024-09-17T23:18:30.647 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:600 IOs, 195191808 bytes 2024-09-17T23:18:30.746 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:700 IOs, 223139840 bytes 2024-09-17T23:18:30.848 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:800 IOs, 251421696 bytes 2024-09-17T23:18:30.984 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:900 IOs, 306545664 bytes 2024-09-17T23:18:31.137 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1000 IOs, 386194432 bytes 2024-09-17T23:18:31.301 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1100 IOs, 486188032 bytes 2024-09-17T23:18:31.435 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1200 IOs, 557115392 bytes 2024-09-17T23:18:31.556 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1300 IOs, 602166272 bytes 2024-09-17T23:18:31.691 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1400 IOs, 668053504 bytes 2024-09-17T23:18:31.827 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1500 IOs, 739803136 bytes 2024-09-17T23:18:31.978 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1600 IOs, 816528384 bytes 2024-09-17T23:18:32.132 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1700 IOs, 917245952 bytes 2024-09-17T23:18:32.272 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1800 IOs, 968551424 bytes 2024-09-17T23:18:32.386 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1900 IOs, 1009866752 bytes 2024-09-17T23:18:32.484 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2000 IOs, 1035627520 bytes 2024-09-17T23:18:32.590 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2100 IOs, 1064543232 bytes 2024-09-17T23:18:32.683 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2200 IOs, 1086941184 bytes 2024-09-17T23:18:32.768 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2300 IOs, 1095620608 bytes 2024-09-17T23:18:32.917 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2400 IOs, 1172392960 bytes 2024-09-17T23:18:33.026 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2500 IOs, 1218606080 bytes 2024-09-17T23:18:33.133 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2600 IOs, 1245806592 bytes 2024-09-17T23:18:33.231 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2700 IOs, 1272203264 bytes 2024-09-17T23:18:33.351 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2800 IOs, 1321827328 bytes 2024-09-17T23:18:33.447 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2900 IOs, 1356440576 bytes 2024-09-17T23:18:33.564 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3000 IOs, 1387853824 bytes 2024-09-17T23:18:33.674 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3100 IOs, 1426425856 bytes 2024-09-17T23:18:33.789 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3200 IOs, 1463783424 bytes 2024-09-17T23:18:33.891 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3300 IOs, 1504820224 bytes 2024-09-17T23:18:33.987 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3400 IOs, 1513633792 bytes 2024-09-17T23:18:34.102 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3500 IOs, 1565649920 bytes 2024-09-17T23:18:34.206 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3600 IOs, 1586071552 bytes 2024-09-17T23:18:34.286 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3700 IOs, 1594667008 bytes 2024-09-17T23:18:34.432 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3800 IOs, 1668620288 bytes 2024-09-17T23:18:34.542 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3900 IOs, 1694232576 bytes 2024-09-17T23:18:34.634 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4000 IOs, 1714377728 bytes 2024-09-17T23:18:34.771 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4100 IOs, 1794231296 bytes 2024-09-17T23:18:34.903 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4200 IOs, 1834592256 bytes 2024-09-17T23:18:35.016 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4300 IOs, 1873891328 bytes 2024-09-17T23:18:35.133 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4400 IOs, 1921486848 bytes 2024-09-17T23:18:35.242 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4500 IOs, 1959374848 bytes 2024-09-17T23:18:35.373 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4600 IOs, 2027555840 bytes 2024-09-17T23:18:35.461 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4700 IOs, 2028166144 bytes 2024-09-17T23:18:35.559 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4800 IOs, 2053985280 bytes 2024-09-17T23:18:35.641 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4900 IOs, 2054596608 bytes 2024-09-17T23:18:35.793 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5000 IOs, 2138770432 bytes 2024-09-17T23:18:35.881 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5100 IOs, 2152184832 bytes 2024-09-17T23:18:36.002 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5200 IOs, 2203040768 bytes 2024-09-17T23:18:36.092 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5300 IOs, 2226152448 bytes 2024-09-17T23:18:36.196 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5400 IOs, 2251893760 bytes 2024-09-17T23:18:36.291 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5500 IOs, 2294439936 bytes 2024-09-17T23:18:36.425 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5600 IOs, 2341437440 bytes 2024-09-17T23:18:36.558 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5700 IOs, 2393257984 bytes 2024-09-17T23:18:36.700 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5800 IOs, 2465636352 bytes 2024-09-17T23:18:36.796 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5900 IOs, 2491015168 bytes 2024-09-17T23:18:36.889 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6000 IOs, 2504384512 bytes 2024-09-17T23:18:36.985 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6100 IOs, 2523652096 bytes 2024-09-17T23:18:37.096 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6200 IOs, 2580942848 bytes 2024-09-17T23:18:37.223 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6300 IOs, 2619181056 bytes 2024-09-17T23:18:37.362 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6400 IOs, 2687266816 bytes 2024-09-17T23:18:37.503 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6500 IOs, 2758013952 bytes 2024-09-17T23:18:37.602 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6600 IOs, 2781818880 bytes 2024-09-17T23:18:37.718 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6700 IOs, 2827304960 bytes 2024-09-17T23:18:37.833 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6800 IOs, 2871446528 bytes 2024-09-17T23:18:37.925 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6900 IOs, 2890189824 bytes 2024-09-17T23:18:38.084 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7000 IOs, 2976352256 bytes 2024-09-17T23:18:38.194 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7100 IOs, 3013454848 bytes 2024-09-17T23:18:38.330 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7200 IOs, 3080312832 bytes 2024-09-17T23:18:38.492 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7300 IOs, 3175089152 bytes 2024-09-17T23:18:38.600 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7400 IOs, 3204699136 bytes 2024-09-17T23:18:38.674 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7500 IOs, 3209180160 bytes 2024-09-17T23:18:38.805 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7600 IOs, 3263158272 bytes 2024-09-17T23:18:38.901 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7700 IOs, 3287168000 bytes 2024-09-17T23:18:39.024 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7800 IOs, 3341421568 bytes 2024-09-17T23:18:39.117 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7900 IOs, 3361491968 bytes 2024-09-17T23:18:39.285 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8000 IOs, 3455933440 bytes 2024-09-17T23:18:39.418 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8100 IOs, 3523205120 bytes 2024-09-17T23:18:39.556 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8200 IOs, 3584311296 bytes 2024-09-17T23:18:39.665 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8300 IOs, 3619385344 bytes 2024-09-17T23:18:39.764 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8400 IOs, 3644552192 bytes 2024-09-17T23:18:39.845 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8500 IOs, 3651468288 bytes 2024-09-17T23:18:39.951 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ error_code=124 2024-09-17T23:18:39.951 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set -e 2024-09-17T23:18:39.951 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' 124 -eq 124 ']' 2024-09-17T23:18:39.951 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:18:39.951 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:18:39.951 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_15 2024-09-17T23:18:39.951 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_image_and_enable_mirror cluster2 mirror image_15 snapshot 128M 2024-09-17T23:18:39.951 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:18:39.951 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:18:39.951 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:18:39.951 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:18:39.952 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_15 2024-09-17T23:18:39.952 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:18:39.952 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=snapshot 2024-09-17T23:18:39.952 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n snapshot ']' 2024-09-17T23:18:39.952 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:18:39.952 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_image cluster2 mirror image_15 128M 2024-09-17T23:18:39.952 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:18:39.952 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:18:39.952 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:18:39.952 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:18:39.952 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_15 2024-09-17T23:18:39.952 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:18:39.952 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local size=128 2024-09-17T23:18:39.953 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n 128M ']' 2024-09-17T23:18:39.953 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ size=128M 2024-09-17T23:18:39.953 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:18:39.953 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_15 2024-09-17T23:18:40.002 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ is_pool_mirror_mode_image mirror 2024-09-17T23:18:40.003 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:18:40.003 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' image = image ']' 2024-09-17T23:18:40.003 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:18:40.003 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ enable_mirror cluster2 mirror image_15 snapshot 2024-09-17T23:18:40.003 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:18:40.003 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:18:40.003 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_15 2024-09-17T23:18:40.003 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=snapshot 2024-09-17T23:18:40.003 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_15 snapshot 2024-09-17T23:18:40.918 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Mirroring enabled 2024-09-17T23:18:40.930 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 info mirror/image_15 2024-09-17T23:18:40.970 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'image_15': 2024-09-17T23:18:40.970 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 128 MiB in 32 objects 2024-09-17T23:18:40.970 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:18:40.970 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 1 2024-09-17T23:18:40.970 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 1326f2b0f6b0 2024-09-17T23:18:40.970 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.1326f2b0f6b0 2024-09-17T23:18:40.970 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:18:40.971 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-17T23:18:40.971 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:18:40.971 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:18:40.971 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:18:39 2024 2024-09-17T23:18:40.971 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:18:39 2024 2024-09-17T23:18:40.971 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:18:39 2024 2024-09-17T23:18:40.971 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:18:40.971 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:18:40.971 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: c5adf338-7077-4024-ab02-6a785842239c 2024-09-17T23:18:40.971 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: true 2024-09-17T23:18:40.973 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n '' ']' 2024-09-17T23:18:40.974 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ stress_write_image cluster2 mirror image_15 2024-09-17T23:18:40.974 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:18:40.974 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:18:40.974 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_15 2024-09-17T23:18:40.974 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-17T23:18:40.976 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local duration=12 2024-09-17T23:18:40.976 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set +e 2024-09-17T23:18:40.976 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_15 --debug-rbd=20 --debug-journaler=20 2024-09-17T23:18:41.200 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:100 IOs, 69980160 bytes 2024-09-17T23:18:41.315 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:200 IOs, 115728384 bytes 2024-09-17T23:18:41.446 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:300 IOs, 180751360 bytes 2024-09-17T23:18:41.559 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:400 IOs, 232660992 bytes 2024-09-17T23:18:41.658 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:500 IOs, 255227904 bytes 2024-09-17T23:18:41.764 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:600 IOs, 288521216 bytes 2024-09-17T23:18:41.879 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:700 IOs, 331095040 bytes 2024-09-17T23:18:41.989 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:800 IOs, 379847680 bytes 2024-09-17T23:18:42.214 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:900 IOs, 539070464 bytes 2024-09-17T23:18:42.302 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1000 IOs, 555084800 bytes 2024-09-17T23:18:42.411 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1100 IOs, 593273856 bytes 2024-09-17T23:18:42.506 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1200 IOs, 616259584 bytes 2024-09-17T23:18:42.595 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1300 IOs, 634724352 bytes 2024-09-17T23:18:42.719 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1400 IOs, 682857472 bytes 2024-09-17T23:18:42.808 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1500 IOs, 697188352 bytes 2024-09-17T23:18:42.889 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1600 IOs, 703638528 bytes 2024-09-17T23:18:43.006 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1700 IOs, 737776640 bytes 2024-09-17T23:18:43.112 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1800 IOs, 768222208 bytes 2024-09-17T23:18:43.198 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1900 IOs, 802586624 bytes 2024-09-17T23:18:43.297 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2000 IOs, 816500736 bytes 2024-09-17T23:18:43.401 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2100 IOs, 837257216 bytes 2024-09-17T23:18:43.492 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2200 IOs, 854347776 bytes 2024-09-17T23:18:43.593 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2300 IOs, 880541696 bytes 2024-09-17T23:18:43.672 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2400 IOs, 903122944 bytes 2024-09-17T23:18:43.793 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2500 IOs, 931207168 bytes 2024-09-17T23:18:43.911 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2600 IOs, 975062016 bytes 2024-09-17T23:18:44.011 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2700 IOs, 998920192 bytes 2024-09-17T23:18:44.128 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2800 IOs, 1038692352 bytes 2024-09-17T23:18:44.218 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2900 IOs, 1056817152 bytes 2024-09-17T23:18:44.344 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3000 IOs, 1118136320 bytes 2024-09-17T23:18:44.437 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3100 IOs, 1130932224 bytes 2024-09-17T23:18:44.513 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3200 IOs, 1132414976 bytes 2024-09-17T23:18:44.626 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3300 IOs, 1171965952 bytes 2024-09-17T23:18:44.740 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3400 IOs, 1211528192 bytes 2024-09-17T23:18:44.835 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3500 IOs, 1233995776 bytes 2024-09-17T23:18:44.930 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3600 IOs, 1251204096 bytes 2024-09-17T23:18:45.124 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3700 IOs, 1371855872 bytes 2024-09-17T23:18:45.227 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3800 IOs, 1410066432 bytes 2024-09-17T23:18:45.359 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3900 IOs, 1465744384 bytes 2024-09-17T23:18:45.449 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4000 IOs, 1481504768 bytes 2024-09-17T23:18:45.537 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4100 IOs, 1492922368 bytes 2024-09-17T23:18:45.646 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4200 IOs, 1527128064 bytes 2024-09-17T23:18:45.761 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4300 IOs, 1572469760 bytes 2024-09-17T23:18:45.846 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4400 IOs, 1583639552 bytes 2024-09-17T23:18:46.002 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4500 IOs, 1659510784 bytes 2024-09-17T23:18:46.127 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4600 IOs, 1721824256 bytes 2024-09-17T23:18:46.274 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4700 IOs, 1783492608 bytes 2024-09-17T23:18:46.358 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4800 IOs, 1794581504 bytes 2024-09-17T23:18:46.468 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4900 IOs, 1852278784 bytes 2024-09-17T23:18:46.585 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5000 IOs, 1879418880 bytes 2024-09-17T23:18:46.702 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5100 IOs, 1920316416 bytes 2024-09-17T23:18:46.830 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5200 IOs, 1975613440 bytes 2024-09-17T23:18:46.937 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5300 IOs, 2007841792 bytes 2024-09-17T23:18:47.066 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5400 IOs, 2060268544 bytes 2024-09-17T23:18:47.204 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5500 IOs, 2120985600 bytes 2024-09-17T23:18:47.288 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5600 IOs, 2126158848 bytes 2024-09-17T23:18:47.384 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5700 IOs, 2151578624 bytes 2024-09-17T23:18:47.463 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5800 IOs, 2152176640 bytes 2024-09-17T23:18:47.571 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5900 IOs, 2202937344 bytes 2024-09-17T23:18:47.679 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6000 IOs, 2225174528 bytes 2024-09-17T23:18:47.767 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6100 IOs, 2238304256 bytes 2024-09-17T23:18:47.948 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6200 IOs, 2344504320 bytes 2024-09-17T23:18:48.070 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6300 IOs, 2394990592 bytes 2024-09-17T23:18:48.181 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6400 IOs, 2434689024 bytes 2024-09-17T23:18:48.264 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6500 IOs, 2435312640 bytes 2024-09-17T23:18:48.388 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6600 IOs, 2484751360 bytes 2024-09-17T23:18:48.510 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6700 IOs, 2543006720 bytes 2024-09-17T23:18:48.618 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6800 IOs, 2575716352 bytes 2024-09-17T23:18:48.718 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6900 IOs, 2597555200 bytes 2024-09-17T23:18:48.810 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7000 IOs, 2609719296 bytes 2024-09-17T23:18:48.907 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7100 IOs, 2637638656 bytes 2024-09-17T23:18:49.060 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7200 IOs, 2711437312 bytes 2024-09-17T23:18:49.197 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7300 IOs, 2781002752 bytes 2024-09-17T23:18:49.352 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7400 IOs, 2844082176 bytes 2024-09-17T23:18:49.457 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7500 IOs, 2879390720 bytes 2024-09-17T23:18:49.553 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7600 IOs, 2899968000 bytes 2024-09-17T23:18:49.667 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7700 IOs, 2938333184 bytes 2024-09-17T23:18:49.785 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7800 IOs, 2984466432 bytes 2024-09-17T23:18:49.877 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7900 IOs, 2997608448 bytes 2024-09-17T23:18:49.978 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8000 IOs, 3023342592 bytes 2024-09-17T23:18:50.091 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8100 IOs, 3060156416 bytes 2024-09-17T23:18:50.196 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8200 IOs, 3087815680 bytes 2024-09-17T23:18:50.297 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8300 IOs, 3113580544 bytes 2024-09-17T23:18:50.425 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8400 IOs, 3179021312 bytes 2024-09-17T23:18:50.526 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8500 IOs, 3185714176 bytes 2024-09-17T23:18:50.655 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8600 IOs, 3246352384 bytes 2024-09-17T23:18:50.788 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8700 IOs, 3300598784 bytes 2024-09-17T23:18:50.911 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8800 IOs, 3343196160 bytes 2024-09-17T23:18:51.027 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8900 IOs, 3387163648 bytes 2024-09-17T23:18:51.161 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9000 IOs, 3436012544 bytes 2024-09-17T23:18:51.271 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9100 IOs, 3472318464 bytes 2024-09-17T23:18:51.393 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9200 IOs, 3522547712 bytes 2024-09-17T23:18:51.514 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9300 IOs, 3569637376 bytes 2024-09-17T23:18:51.635 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9400 IOs, 3618057216 bytes 2024-09-17T23:18:51.755 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9500 IOs, 3661598720 bytes 2024-09-17T23:18:51.867 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9600 IOs, 3701046272 bytes 2024-09-17T23:18:52.013 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9700 IOs, 3773277184 bytes 2024-09-17T23:18:52.093 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9800 IOs, 3774124032 bytes 2024-09-17T23:18:52.214 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9900 IOs, 3820910592 bytes 2024-09-17T23:18:52.290 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10000 IOs, 3821522944 bytes 2024-09-17T23:18:52.457 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10100 IOs, 3929251840 bytes 2024-09-17T23:18:52.632 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10200 IOs, 4032186368 bytes 2024-09-17T23:18:52.741 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10300 IOs, 4053956608 bytes 2024-09-17T23:18:52.853 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10400 IOs, 4086557696 bytes 2024-09-17T23:18:52.985 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ error_code=124 2024-09-17T23:18:52.985 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set -e 2024-09-17T23:18:52.986 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' 124 -eq 124 ']' 2024-09-17T23:18:52.986 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:18:52.986 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:18:52.986 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_16 2024-09-17T23:18:52.986 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_image_and_enable_mirror cluster2 mirror image_16 snapshot 128M 2024-09-17T23:18:52.986 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:18:52.986 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:18:52.986 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:18:52.986 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:18:52.986 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_16 2024-09-17T23:18:52.986 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:18:52.986 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=snapshot 2024-09-17T23:18:52.986 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n snapshot ']' 2024-09-17T23:18:52.986 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:18:52.986 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_image cluster2 mirror image_16 128M 2024-09-17T23:18:52.987 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:18:52.987 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:18:52.987 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:18:52.987 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:18:52.987 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_16 2024-09-17T23:18:52.987 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:18:52.987 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local size=128 2024-09-17T23:18:52.987 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n 128M ']' 2024-09-17T23:18:52.987 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ size=128M 2024-09-17T23:18:52.987 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:18:52.987 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_16 2024-09-17T23:18:53.038 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ is_pool_mirror_mode_image mirror 2024-09-17T23:18:53.038 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:18:53.038 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' image = image ']' 2024-09-17T23:18:53.039 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:18:53.039 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ enable_mirror cluster2 mirror image_16 snapshot 2024-09-17T23:18:53.039 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:18:53.039 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:18:53.039 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_16 2024-09-17T23:18:53.039 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=snapshot 2024-09-17T23:18:53.039 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_16 snapshot 2024-09-17T23:18:53.948 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Mirroring enabled 2024-09-17T23:18:53.960 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 info mirror/image_16 2024-09-17T23:18:54.001 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'image_16': 2024-09-17T23:18:54.001 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 128 MiB in 32 objects 2024-09-17T23:18:54.001 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:18:54.001 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 1 2024-09-17T23:18:54.001 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 1332915b2ea5 2024-09-17T23:18:54.001 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.1332915b2ea5 2024-09-17T23:18:54.001 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:18:54.001 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-17T23:18:54.001 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:18:54.002 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:18:54.002 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:18:53 2024 2024-09-17T23:18:54.002 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:18:53 2024 2024-09-17T23:18:54.002 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:18:53 2024 2024-09-17T23:18:54.002 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:18:54.002 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:18:54.002 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: b0d8f9e8-4e5b-45cd-b488-325c8908f543 2024-09-17T23:18:54.002 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: true 2024-09-17T23:18:54.004 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n '' ']' 2024-09-17T23:18:54.005 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ stress_write_image cluster2 mirror image_16 2024-09-17T23:18:54.005 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:18:54.005 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:18:54.005 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_16 2024-09-17T23:18:54.005 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-17T23:18:54.007 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local duration=13 2024-09-17T23:18:54.007 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set +e 2024-09-17T23:18:54.007 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_16 --debug-rbd=20 --debug-journaler=20 2024-09-17T23:18:54.424 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:100 IOs, 58959872 bytes 2024-09-17T23:18:54.559 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:200 IOs, 119646208 bytes 2024-09-17T23:18:54.635 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:300 IOs, 120270848 bytes 2024-09-17T23:18:54.735 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:400 IOs, 146292736 bytes 2024-09-17T23:18:54.829 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:500 IOs, 166567936 bytes 2024-09-17T23:18:54.951 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:600 IOs, 215239680 bytes 2024-09-17T23:18:55.067 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:700 IOs, 256116736 bytes 2024-09-17T23:18:55.182 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:800 IOs, 293881856 bytes 2024-09-17T23:18:55.319 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:900 IOs, 357689344 bytes 2024-09-17T23:18:55.410 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1000 IOs, 372515840 bytes 2024-09-17T23:18:55.498 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1100 IOs, 374516736 bytes 2024-09-17T23:18:55.581 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1200 IOs, 382695424 bytes 2024-09-17T23:18:55.679 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1300 IOs, 409372672 bytes 2024-09-17T23:18:55.833 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1400 IOs, 490834944 bytes 2024-09-17T23:18:55.947 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1500 IOs, 524154880 bytes 2024-09-17T23:18:56.084 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1600 IOs, 584226816 bytes 2024-09-17T23:18:56.189 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1700 IOs, 607355904 bytes 2024-09-17T23:18:56.349 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1800 IOs, 697633792 bytes 2024-09-17T23:18:56.449 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1900 IOs, 721353728 bytes 2024-09-17T23:18:56.550 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2000 IOs, 754702336 bytes 2024-09-17T23:18:56.644 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2100 IOs, 770025472 bytes 2024-09-17T23:18:56.737 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2200 IOs, 800149504 bytes 2024-09-17T23:18:56.853 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2300 IOs, 827385856 bytes 2024-09-17T23:18:57.001 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2400 IOs, 886410240 bytes 2024-09-17T23:18:57.148 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2500 IOs, 959376384 bytes 2024-09-17T23:18:57.281 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2600 IOs, 1015545856 bytes 2024-09-17T23:18:57.356 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2700 IOs, 1039638528 bytes 2024-09-17T23:18:57.456 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2800 IOs, 1040260096 bytes 2024-09-17T23:18:57.599 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2900 IOs, 1112360960 bytes 2024-09-17T23:18:57.691 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3000 IOs, 1131403264 bytes 2024-09-17T23:18:57.783 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3100 IOs, 1152707584 bytes 2024-09-17T23:18:57.871 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3200 IOs, 1164705792 bytes 2024-09-17T23:18:57.964 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3300 IOs, 1185157120 bytes 2024-09-17T23:18:58.083 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3400 IOs, 1220432896 bytes 2024-09-17T23:18:58.254 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3500 IOs, 1324530688 bytes 2024-09-17T23:18:58.355 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3600 IOs, 1345415168 bytes 2024-09-17T23:18:58.472 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3700 IOs, 1373657088 bytes 2024-09-17T23:18:58.576 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3800 IOs, 1400168448 bytes 2024-09-17T23:18:58.756 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3900 IOs, 1503157248 bytes 2024-09-17T23:18:58.848 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4000 IOs, 1548991488 bytes 2024-09-17T23:18:58.993 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4100 IOs, 1597356032 bytes 2024-09-17T23:18:59.141 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4200 IOs, 1689641984 bytes 2024-09-17T23:18:59.255 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4300 IOs, 1709371392 bytes 2024-09-17T23:18:59.424 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4400 IOs, 1811318784 bytes 2024-09-17T23:18:59.543 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4500 IOs, 1851808768 bytes 2024-09-17T23:18:59.663 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4600 IOs, 1903575040 bytes 2024-09-17T23:18:59.797 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4700 IOs, 1960956928 bytes 2024-09-17T23:18:59.910 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4800 IOs, 2004367360 bytes 2024-09-17T23:19:00.018 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4900 IOs, 2040005632 bytes 2024-09-17T23:19:00.166 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5000 IOs, 2106971136 bytes 2024-09-17T23:19:00.262 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5100 IOs, 2125447168 bytes 2024-09-17T23:19:00.359 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5200 IOs, 2145321984 bytes 2024-09-17T23:19:00.464 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5300 IOs, 2171453440 bytes 2024-09-17T23:19:00.597 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5400 IOs, 2241808384 bytes 2024-09-17T23:19:00.811 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5500 IOs, 2292106240 bytes 2024-09-17T23:19:00.941 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5600 IOs, 2349935616 bytes 2024-09-17T23:19:01.066 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5700 IOs, 2393032704 bytes 2024-09-17T23:19:01.214 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5800 IOs, 2465741824 bytes 2024-09-17T23:19:01.350 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5900 IOs, 2525680640 bytes 2024-09-17T23:19:01.485 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6000 IOs, 2582846464 bytes 2024-09-17T23:19:01.614 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6100 IOs, 2607549440 bytes 2024-09-17T23:19:01.754 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6200 IOs, 2672030720 bytes 2024-09-17T23:19:01.843 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6300 IOs, 2693696512 bytes 2024-09-17T23:19:01.946 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6400 IOs, 2710282240 bytes 2024-09-17T23:19:02.051 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6500 IOs, 2729815040 bytes 2024-09-17T23:19:02.155 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6600 IOs, 2756966400 bytes 2024-09-17T23:19:02.261 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6700 IOs, 2785779712 bytes 2024-09-17T23:19:02.344 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6800 IOs, 2789220352 bytes 2024-09-17T23:19:02.452 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6900 IOs, 2821182464 bytes 2024-09-17T23:19:02.541 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7000 IOs, 2835714048 bytes 2024-09-17T23:19:02.686 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7100 IOs, 2901903360 bytes 2024-09-17T23:19:02.786 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7200 IOs, 2926350336 bytes 2024-09-17T23:19:02.870 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7300 IOs, 2929343488 bytes 2024-09-17T23:19:02.977 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7400 IOs, 2959790080 bytes 2024-09-17T23:19:03.076 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7500 IOs, 2976686080 bytes 2024-09-17T23:19:03.244 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7600 IOs, 3080934400 bytes 2024-09-17T23:19:03.364 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7700 IOs, 3113363456 bytes 2024-09-17T23:19:03.473 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7800 IOs, 3146770432 bytes 2024-09-17T23:19:03.778 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7900 IOs, 3176453120 bytes 2024-09-17T23:19:03.891 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8000 IOs, 3211616256 bytes 2024-09-17T23:19:04.028 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8100 IOs, 3275204608 bytes 2024-09-17T23:19:04.124 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8200 IOs, 3307895808 bytes 2024-09-17T23:19:04.275 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8300 IOs, 3374960640 bytes 2024-09-17T23:19:04.374 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8400 IOs, 3382446080 bytes 2024-09-17T23:19:04.504 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8500 IOs, 3435534336 bytes 2024-09-17T23:19:04.612 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8600 IOs, 3468897280 bytes 2024-09-17T23:19:04.738 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8700 IOs, 3522210816 bytes 2024-09-17T23:19:04.864 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8800 IOs, 3574194176 bytes 2024-09-17T23:19:04.961 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8900 IOs, 3591164928 bytes 2024-09-17T23:19:05.045 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9000 IOs, 3594704896 bytes 2024-09-17T23:19:05.136 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9100 IOs, 3605779456 bytes 2024-09-17T23:19:05.259 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9200 IOs, 3655695360 bytes 2024-09-17T23:19:05.365 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9300 IOs, 3683360768 bytes 2024-09-17T23:19:05.465 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9400 IOs, 3694844928 bytes 2024-09-17T23:19:05.564 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9500 IOs, 3712217088 bytes 2024-09-17T23:19:05.666 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9600 IOs, 3735721984 bytes 2024-09-17T23:19:05.766 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9700 IOs, 3758627840 bytes 2024-09-17T23:19:05.925 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9800 IOs, 3795648512 bytes 2024-09-17T23:19:06.082 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9900 IOs, 3866524672 bytes 2024-09-17T23:19:06.209 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10000 IOs, 3917402112 bytes 2024-09-17T23:19:06.306 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10100 IOs, 3936471040 bytes 2024-09-17T23:19:06.422 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10200 IOs, 3983481856 bytes 2024-09-17T23:19:06.588 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10300 IOs, 4066020352 bytes 2024-09-17T23:19:06.674 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10400 IOs, 4074995712 bytes 2024-09-17T23:19:06.830 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10500 IOs, 4157065216 bytes 2024-09-17T23:19:06.973 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10600 IOs, 4227336192 bytes 2024-09-17T23:19:07.012 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ error_code=124 2024-09-17T23:19:07.012 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set -e 2024-09-17T23:19:07.012 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' 124 -eq 124 ']' 2024-09-17T23:19:07.013 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:19:07.013 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:19:07.013 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_17 2024-09-17T23:19:07.013 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_image_and_enable_mirror cluster2 mirror image_17 snapshot 128M 2024-09-17T23:19:07.013 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:19:07.013 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:19:07.013 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:19:07.013 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:19:07.013 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_17 2024-09-17T23:19:07.013 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:19:07.013 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=snapshot 2024-09-17T23:19:07.013 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n snapshot ']' 2024-09-17T23:19:07.014 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:19:07.014 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_image cluster2 mirror image_17 128M 2024-09-17T23:19:07.014 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:19:07.014 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:19:07.014 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:19:07.015 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:19:07.015 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_17 2024-09-17T23:19:07.015 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:19:07.015 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local size=128 2024-09-17T23:19:07.015 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n 128M ']' 2024-09-17T23:19:07.015 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ size=128M 2024-09-17T23:19:07.015 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:19:07.015 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_17 2024-09-17T23:19:07.062 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ is_pool_mirror_mode_image mirror 2024-09-17T23:19:07.063 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:19:07.063 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' image = image ']' 2024-09-17T23:19:07.063 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:19:07.063 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ enable_mirror cluster2 mirror image_17 snapshot 2024-09-17T23:19:07.063 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:19:07.063 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:19:07.063 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_17 2024-09-17T23:19:07.063 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=snapshot 2024-09-17T23:19:07.063 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_17 snapshot 2024-09-17T23:19:07.972 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Mirroring enabled 2024-09-17T23:19:07.984 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 info mirror/image_17 2024-09-17T23:19:08.024 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'image_17': 2024-09-17T23:19:08.024 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 128 MiB in 32 objects 2024-09-17T23:19:08.025 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:19:08.025 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 1 2024-09-17T23:19:08.025 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 133ec93c649a 2024-09-17T23:19:08.025 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.133ec93c649a 2024-09-17T23:19:08.025 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:19:08.025 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-17T23:19:08.025 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:19:08.025 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:19:08.025 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:19:07 2024 2024-09-17T23:19:08.025 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:19:07 2024 2024-09-17T23:19:08.025 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:19:07 2024 2024-09-17T23:19:08.025 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:19:08.025 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:19:08.025 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: 1c42b8ab-b2d2-4388-b7b5-b3e796c582f2 2024-09-17T23:19:08.026 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: true 2024-09-17T23:19:08.028 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n '' ']' 2024-09-17T23:19:08.028 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ stress_write_image cluster2 mirror image_17 2024-09-17T23:19:08.028 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:19:08.028 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:19:08.028 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_17 2024-09-17T23:19:08.028 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-17T23:19:08.030 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local duration=8 2024-09-17T23:19:08.030 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set +e 2024-09-17T23:19:08.030 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_17 --debug-rbd=20 --debug-journaler=20 2024-09-17T23:19:08.232 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:100 IOs, 33405952 bytes 2024-09-17T23:19:08.312 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:200 IOs, 34041856 bytes 2024-09-17T23:19:08.455 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:300 IOs, 104153088 bytes 2024-09-17T23:19:08.581 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:400 IOs, 169850880 bytes 2024-09-17T23:19:08.673 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:500 IOs, 178721792 bytes 2024-09-17T23:19:08.835 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:600 IOs, 267742208 bytes 2024-09-17T23:19:08.935 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:700 IOs, 294541312 bytes 2024-09-17T23:19:09.068 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:800 IOs, 356464640 bytes 2024-09-17T23:19:09.179 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:900 IOs, 394109952 bytes 2024-09-17T23:19:09.299 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1000 IOs, 467055616 bytes 2024-09-17T23:19:09.445 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1100 IOs, 512828416 bytes 2024-09-17T23:19:09.582 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1200 IOs, 582591488 bytes 2024-09-17T23:19:09.685 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1300 IOs, 604449792 bytes 2024-09-17T23:19:09.783 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1400 IOs, 628126720 bytes 2024-09-17T23:19:09.884 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1500 IOs, 654966784 bytes 2024-09-17T23:19:09.975 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1600 IOs, 667942912 bytes 2024-09-17T23:19:10.062 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1700 IOs, 669521920 bytes 2024-09-17T23:19:10.199 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1800 IOs, 754210816 bytes 2024-09-17T23:19:10.377 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1900 IOs, 830344192 bytes 2024-09-17T23:19:10.475 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2000 IOs, 859930624 bytes 2024-09-17T23:19:10.602 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2100 IOs, 893233152 bytes 2024-09-17T23:19:10.743 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2200 IOs, 975562752 bytes 2024-09-17T23:19:10.913 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2300 IOs, 1060722688 bytes 2024-09-17T23:19:11.083 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2400 IOs, 1154099200 bytes 2024-09-17T23:19:11.243 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2500 IOs, 1243338752 bytes 2024-09-17T23:19:11.414 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2600 IOs, 1335312384 bytes 2024-09-17T23:19:11.525 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2700 IOs, 1365065728 bytes 2024-09-17T23:19:11.657 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2800 IOs, 1422760960 bytes 2024-09-17T23:19:11.751 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2900 IOs, 1439139840 bytes 2024-09-17T23:19:11.838 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3000 IOs, 1449033728 bytes 2024-09-17T23:19:12.014 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3100 IOs, 1553114112 bytes 2024-09-17T23:19:12.118 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3200 IOs, 1581947904 bytes 2024-09-17T23:19:12.239 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3300 IOs, 1627318272 bytes 2024-09-17T23:19:12.335 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3400 IOs, 1645576192 bytes 2024-09-17T23:19:12.469 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3500 IOs, 1699335168 bytes 2024-09-17T23:19:12.573 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3600 IOs, 1738029056 bytes 2024-09-17T23:19:12.694 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3700 IOs, 1774176256 bytes 2024-09-17T23:19:12.827 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3800 IOs, 1838952448 bytes 2024-09-17T23:19:12.957 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3900 IOs, 1891190784 bytes 2024-09-17T23:19:13.090 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4000 IOs, 1966697472 bytes 2024-09-17T23:19:13.249 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4100 IOs, 2017841152 bytes 2024-09-17T23:19:13.405 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4200 IOs, 2089610240 bytes 2024-09-17T23:19:13.556 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4300 IOs, 2178067456 bytes 2024-09-17T23:19:13.668 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4400 IOs, 2205383680 bytes 2024-09-17T23:19:13.768 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4500 IOs, 2228403200 bytes 2024-09-17T23:19:13.861 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4600 IOs, 2243538944 bytes 2024-09-17T23:19:13.971 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4700 IOs, 2278365184 bytes 2024-09-17T23:19:14.049 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4800 IOs, 2289131520 bytes 2024-09-17T23:19:14.195 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4900 IOs, 2349227008 bytes 2024-09-17T23:19:14.285 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5000 IOs, 2361186304 bytes 2024-09-17T23:19:14.398 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5100 IOs, 2400606208 bytes 2024-09-17T23:19:14.513 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5200 IOs, 2432371712 bytes 2024-09-17T23:19:14.614 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5300 IOs, 2454282240 bytes 2024-09-17T23:19:14.760 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5400 IOs, 2518210560 bytes 2024-09-17T23:19:14.875 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5500 IOs, 2557377536 bytes 2024-09-17T23:19:15.053 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5600 IOs, 2656862208 bytes 2024-09-17T23:19:15.153 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5700 IOs, 2680593408 bytes 2024-09-17T23:19:15.244 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5800 IOs, 2694895616 bytes 2024-09-17T23:19:15.342 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5900 IOs, 2714440704 bytes 2024-09-17T23:19:15.432 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6000 IOs, 2728296448 bytes 2024-09-17T23:19:15.548 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6100 IOs, 2763890688 bytes 2024-09-17T23:19:15.638 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6200 IOs, 2775476224 bytes 2024-09-17T23:19:15.731 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6300 IOs, 2788634624 bytes 2024-09-17T23:19:15.861 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6400 IOs, 2847672320 bytes 2024-09-17T23:19:15.966 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6500 IOs, 2871987200 bytes 2024-09-17T23:19:16.039 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ error_code=124 2024-09-17T23:19:16.039 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set -e 2024-09-17T23:19:16.039 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' 124 -eq 124 ']' 2024-09-17T23:19:16.039 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:19:16.039 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:19:16.039 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_18 2024-09-17T23:19:16.040 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_image_and_enable_mirror cluster2 mirror image_18 snapshot 128M 2024-09-17T23:19:16.040 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:19:16.040 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:19:16.040 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:19:16.040 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:19:16.040 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_18 2024-09-17T23:19:16.040 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:19:16.040 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=snapshot 2024-09-17T23:19:16.040 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n snapshot ']' 2024-09-17T23:19:16.040 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:19:16.040 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_image cluster2 mirror image_18 128M 2024-09-17T23:19:16.040 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:19:16.040 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:19:16.041 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:19:16.041 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:19:16.041 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_18 2024-09-17T23:19:16.041 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:19:16.041 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local size=128 2024-09-17T23:19:16.041 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n 128M ']' 2024-09-17T23:19:16.041 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ size=128M 2024-09-17T23:19:16.041 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:19:16.041 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_18 2024-09-17T23:19:16.092 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ is_pool_mirror_mode_image mirror 2024-09-17T23:19:16.092 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:19:16.092 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' image = image ']' 2024-09-17T23:19:16.092 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:19:16.092 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ enable_mirror cluster2 mirror image_18 snapshot 2024-09-17T23:19:16.092 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:19:16.092 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:19:16.092 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_18 2024-09-17T23:19:16.092 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=snapshot 2024-09-17T23:19:16.092 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_18 snapshot 2024-09-17T23:19:16.985 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Mirroring enabled 2024-09-17T23:19:16.996 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 info mirror/image_18 2024-09-17T23:19:17.037 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'image_18': 2024-09-17T23:19:17.037 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 128 MiB in 32 objects 2024-09-17T23:19:17.037 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:19:17.037 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 1 2024-09-17T23:19:17.037 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 134a2fbe9d26 2024-09-17T23:19:17.037 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.134a2fbe9d26 2024-09-17T23:19:17.037 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:19:17.037 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-17T23:19:17.037 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:19:17.037 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:19:17.038 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:19:16 2024 2024-09-17T23:19:17.038 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:19:16 2024 2024-09-17T23:19:17.038 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:19:16 2024 2024-09-17T23:19:17.038 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:19:17.038 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:19:17.038 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: 82669132-31fe-4405-bf71-3a31515c863e 2024-09-17T23:19:17.038 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: true 2024-09-17T23:19:17.040 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n '' ']' 2024-09-17T23:19:17.040 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ stress_write_image cluster2 mirror image_18 2024-09-17T23:19:17.041 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:19:17.041 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:19:17.041 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_18 2024-09-17T23:19:17.041 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-17T23:19:17.042 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local duration=8 2024-09-17T23:19:17.043 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set +e 2024-09-17T23:19:17.043 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_18 --debug-rbd=20 --debug-journaler=20 2024-09-17T23:19:17.217 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:100 IOs, 27432960 bytes 2024-09-17T23:19:17.329 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:200 IOs, 65441792 bytes 2024-09-17T23:19:17.416 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:300 IOs, 76241920 bytes 2024-09-17T23:19:17.543 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:400 IOs, 131078144 bytes 2024-09-17T23:19:17.646 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:500 IOs, 159295488 bytes 2024-09-17T23:19:17.762 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:600 IOs, 201073664 bytes 2024-09-17T23:19:17.882 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:700 IOs, 250199040 bytes 2024-09-17T23:19:17.977 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:800 IOs, 268846080 bytes 2024-09-17T23:19:18.077 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:900 IOs, 295626752 bytes 2024-09-17T23:19:18.185 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1000 IOs, 330507264 bytes 2024-09-17T23:19:18.308 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1100 IOs, 377007104 bytes 2024-09-17T23:19:18.403 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1200 IOs, 397105152 bytes 2024-09-17T23:19:18.517 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1300 IOs, 459329536 bytes 2024-09-17T23:19:18.689 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1400 IOs, 530834432 bytes 2024-09-17T23:19:18.787 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1500 IOs, 556091392 bytes 2024-09-17T23:19:18.893 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1600 IOs, 616237056 bytes 2024-09-17T23:19:19.016 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1700 IOs, 634796032 bytes 2024-09-17T23:19:19.111 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1800 IOs, 653108224 bytes 2024-09-17T23:19:19.218 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1900 IOs, 687369216 bytes 2024-09-17T23:19:19.333 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2000 IOs, 719337472 bytes 2024-09-17T23:19:19.424 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2100 IOs, 732669952 bytes 2024-09-17T23:19:19.578 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2200 IOs, 814905344 bytes 2024-09-17T23:19:19.748 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2300 IOs, 914432000 bytes 2024-09-17T23:19:19.892 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2400 IOs, 989845504 bytes 2024-09-17T23:19:19.921 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.3497772827414717) > weight (0.25) 2024-09-17T23:19:19.922 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.7509584269331024) > weight (0.25) 2024-09-17T23:19:19.922 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2024-09-17T23:19:19.922 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2024-09-17T23:19:19.922 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2024-09-17T23:19:19.924 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi038.stderr:2024-09-17T23:19:19.920+0000 7f4f0779e640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 39781) UID: 1000 2024-09-17T23:19:20.003 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2500 IOs, 1006184448 bytes 2024-09-17T23:19:20.106 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2600 IOs, 1028356096 bytes 2024-09-17T23:19:20.220 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2700 IOs, 1073129472 bytes 2024-09-17T23:19:20.345 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2800 IOs, 1116350464 bytes 2024-09-17T23:19:20.504 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2900 IOs, 1199351808 bytes 2024-09-17T23:19:20.620 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3000 IOs, 1234764800 bytes 2024-09-17T23:19:20.696 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3100 IOs, 1235378176 bytes 2024-09-17T23:19:20.802 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3200 IOs, 1262396416 bytes 2024-09-17T23:19:20.918 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3300 IOs, 1303282688 bytes 2024-09-17T23:19:21.019 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3400 IOs, 1327202304 bytes 2024-09-17T23:19:21.137 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3500 IOs, 1367909376 bytes 2024-09-17T23:19:21.274 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3600 IOs, 1431442432 bytes 2024-09-17T23:19:21.369 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3700 IOs, 1448065024 bytes 2024-09-17T23:19:21.459 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3800 IOs, 1462370304 bytes 2024-09-17T23:19:21.579 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3900 IOs, 1511566336 bytes 2024-09-17T23:19:21.690 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4000 IOs, 1538857984 bytes 2024-09-17T23:19:21.794 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4100 IOs, 1563840512 bytes 2024-09-17T23:19:21.899 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4200 IOs, 1590952960 bytes 2024-09-17T23:19:22.009 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4300 IOs, 1625588736 bytes 2024-09-17T23:19:22.113 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4400 IOs, 1647372288 bytes 2024-09-17T23:19:22.261 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4500 IOs, 1721375744 bytes 2024-09-17T23:19:22.359 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4600 IOs, 1742577664 bytes 2024-09-17T23:19:22.496 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4700 IOs, 1803501568 bytes 2024-09-17T23:19:22.607 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4800 IOs, 1835051008 bytes 2024-09-17T23:19:22.727 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4900 IOs, 1879678976 bytes 2024-09-17T23:19:22.842 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5000 IOs, 1921917952 bytes 2024-09-17T23:19:22.953 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5100 IOs, 1958670336 bytes 2024-09-17T23:19:23.062 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5200 IOs, 1991866368 bytes 2024-09-17T23:19:23.171 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5300 IOs, 2028083200 bytes 2024-09-17T23:19:23.301 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5400 IOs, 2095841280 bytes 2024-09-17T23:19:23.426 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5500 IOs, 2127904768 bytes 2024-09-17T23:19:23.565 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5600 IOs, 2191560704 bytes 2024-09-17T23:19:23.664 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5700 IOs, 2221154304 bytes 2024-09-17T23:19:23.776 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5800 IOs, 2263415808 bytes 2024-09-17T23:19:23.868 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5900 IOs, 2285579264 bytes 2024-09-17T23:19:24.026 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6000 IOs, 2348878848 bytes 2024-09-17T23:19:24.118 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6100 IOs, 2364905472 bytes 2024-09-17T23:19:24.214 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6200 IOs, 2385696768 bytes 2024-09-17T23:19:24.317 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6300 IOs, 2411572224 bytes 2024-09-17T23:19:24.456 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6400 IOs, 2473272320 bytes 2024-09-17T23:19:24.553 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6500 IOs, 2490655744 bytes 2024-09-17T23:19:24.640 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6600 IOs, 2500684800 bytes 2024-09-17T23:19:24.720 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6700 IOs, 2501315584 bytes 2024-09-17T23:19:24.875 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6800 IOs, 2571948032 bytes 2024-09-17T23:19:24.974 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6900 IOs, 2593817600 bytes 2024-09-17T23:19:25.048 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ error_code=124 2024-09-17T23:19:25.049 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set -e 2024-09-17T23:19:25.049 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' 124 -eq 124 ']' 2024-09-17T23:19:25.049 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:19:25.049 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:19:25.049 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_19 2024-09-17T23:19:25.049 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_image_and_enable_mirror cluster2 mirror image_19 snapshot 128M 2024-09-17T23:19:25.049 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:19:25.049 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:19:25.049 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:19:25.049 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:19:25.049 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_19 2024-09-17T23:19:25.049 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:19:25.050 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=snapshot 2024-09-17T23:19:25.050 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n snapshot ']' 2024-09-17T23:19:25.050 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:19:25.050 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_image cluster2 mirror image_19 128M 2024-09-17T23:19:25.050 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:19:25.050 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:19:25.050 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:19:25.050 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:19:25.051 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_19 2024-09-17T23:19:25.051 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:19:25.051 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local size=128 2024-09-17T23:19:25.051 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n 128M ']' 2024-09-17T23:19:25.051 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ size=128M 2024-09-17T23:19:25.051 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:19:25.051 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_19 2024-09-17T23:19:25.099 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ is_pool_mirror_mode_image mirror 2024-09-17T23:19:25.099 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:19:25.099 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' image = image ']' 2024-09-17T23:19:25.099 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:19:25.099 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ enable_mirror cluster2 mirror image_19 snapshot 2024-09-17T23:19:25.099 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:19:25.100 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:19:25.100 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_19 2024-09-17T23:19:25.100 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=snapshot 2024-09-17T23:19:25.100 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_19 snapshot 2024-09-17T23:19:25.923 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2024-09-17T23:19:25.923 INFO:teuthology.orchestra.run:waiting for 600 2024-09-17T23:19:25.923 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2024-09-17T23:19:25.924 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2024-09-17T23:19:25.924 DEBUG:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2024-09-17T23:19:25.927 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2024-09-17T23:19:25.927 INFO:tasks.rbd_mirror_thrash:waiting for 60 secs before thrashing 2024-09-17T23:19:25.997 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Mirroring enabled 2024-09-17T23:19:26.009 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 info mirror/image_19 2024-09-17T23:19:26.059 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'image_19': 2024-09-17T23:19:26.060 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 128 MiB in 32 objects 2024-09-17T23:19:26.060 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:19:26.060 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 1 2024-09-17T23:19:26.060 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 1356ec32456 2024-09-17T23:19:26.060 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.1356ec32456 2024-09-17T23:19:26.060 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:19:26.060 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-17T23:19:26.060 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:19:26.060 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:19:26.060 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:19:25 2024 2024-09-17T23:19:26.060 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:19:25 2024 2024-09-17T23:19:26.061 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:19:25 2024 2024-09-17T23:19:26.061 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:19:26.061 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:19:26.061 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: 263f2d9e-807d-4be1-b51d-f2bd7fb7a5d6 2024-09-17T23:19:26.061 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: true 2024-09-17T23:19:26.063 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n '' ']' 2024-09-17T23:19:26.063 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ stress_write_image cluster2 mirror image_19 2024-09-17T23:19:26.063 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:19:26.063 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:19:26.063 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_19 2024-09-17T23:19:26.063 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-17T23:19:26.065 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local duration=14 2024-09-17T23:19:26.065 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set +e 2024-09-17T23:19:26.065 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_19 --debug-rbd=20 --debug-journaler=20 2024-09-17T23:19:26.312 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:100 IOs, 89468928 bytes 2024-09-17T23:19:26.425 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:200 IOs, 125193216 bytes 2024-09-17T23:19:26.556 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:300 IOs, 189578240 bytes 2024-09-17T23:19:26.693 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:400 IOs, 248124416 bytes 2024-09-17T23:19:26.827 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:500 IOs, 302238720 bytes 2024-09-17T23:19:26.946 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:600 IOs, 337546240 bytes 2024-09-17T23:19:27.084 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:700 IOs, 397952000 bytes 2024-09-17T23:19:27.184 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:800 IOs, 424051712 bytes 2024-09-17T23:19:27.270 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:900 IOs, 434838528 bytes 2024-09-17T23:19:27.418 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1000 IOs, 523008000 bytes 2024-09-17T23:19:27.552 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1100 IOs, 567834624 bytes 2024-09-17T23:19:27.656 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1200 IOs, 597496832 bytes 2024-09-17T23:19:27.739 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1300 IOs, 603017216 bytes 2024-09-17T23:19:27.822 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1400 IOs, 608890880 bytes 2024-09-17T23:19:27.974 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1500 IOs, 690341888 bytes 2024-09-17T23:19:28.087 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1600 IOs, 719346688 bytes 2024-09-17T23:19:28.200 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1700 IOs, 756833280 bytes 2024-09-17T23:19:28.346 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1800 IOs, 824843264 bytes 2024-09-17T23:19:28.441 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1900 IOs, 842283008 bytes 2024-09-17T23:19:28.557 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2000 IOs, 879787008 bytes 2024-09-17T23:19:28.685 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2100 IOs, 930466816 bytes 2024-09-17T23:19:28.799 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2200 IOs, 963921920 bytes 2024-09-17T23:19:28.918 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2300 IOs, 1002222592 bytes 2024-09-17T23:19:29.012 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2400 IOs, 1018102784 bytes 2024-09-17T23:19:29.131 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2500 IOs, 1060127744 bytes 2024-09-17T23:19:29.248 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2600 IOs, 1098260480 bytes 2024-09-17T23:19:29.364 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2700 IOs, 1139161088 bytes 2024-09-17T23:19:29.467 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2800 IOs, 1167514624 bytes 2024-09-17T23:19:29.593 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2900 IOs, 1215470592 bytes 2024-09-17T23:19:29.674 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3000 IOs, 1232171008 bytes 2024-09-17T23:19:29.806 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3100 IOs, 1272211456 bytes 2024-09-17T23:19:29.959 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3200 IOs, 1349979136 bytes 2024-09-17T23:19:30.076 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3300 IOs, 1392066560 bytes 2024-09-17T23:19:30.204 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3400 IOs, 1442153472 bytes 2024-09-17T23:19:30.314 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3500 IOs, 1479407616 bytes 2024-09-17T23:19:30.455 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3600 IOs, 1542066176 bytes 2024-09-17T23:19:30.538 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3700 IOs, 1542684672 bytes 2024-09-17T23:19:30.647 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3800 IOs, 1588448256 bytes 2024-09-17T23:19:30.752 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3900 IOs, 1598210048 bytes 2024-09-17T23:19:30.898 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4000 IOs, 1668212736 bytes 2024-09-17T23:19:30.992 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4100 IOs, 1683533824 bytes 2024-09-17T23:19:31.073 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4200 IOs, 1687418880 bytes 2024-09-17T23:19:31.161 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4300 IOs, 1695464448 bytes 2024-09-17T23:19:31.302 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4400 IOs, 1761931264 bytes 2024-09-17T23:19:31.406 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4500 IOs, 1798507520 bytes 2024-09-17T23:19:31.543 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4600 IOs, 1845665792 bytes 2024-09-17T23:19:31.687 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4700 IOs, 1914691584 bytes 2024-09-17T23:19:31.806 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4800 IOs, 1957192704 bytes 2024-09-17T23:19:31.926 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4900 IOs, 1997776896 bytes 2024-09-17T23:19:32.008 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5000 IOs, 2000888832 bytes 2024-09-17T23:19:32.123 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5100 IOs, 2035742720 bytes 2024-09-17T23:19:32.231 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5200 IOs, 2070778880 bytes 2024-09-17T23:19:32.364 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5300 IOs, 2124908544 bytes 2024-09-17T23:19:32.463 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5400 IOs, 2150103040 bytes 2024-09-17T23:19:32.560 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5500 IOs, 2168303616 bytes 2024-09-17T23:19:32.661 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5600 IOs, 2187708416 bytes 2024-09-17T23:19:32.797 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5700 IOs, 2259486720 bytes 2024-09-17T23:19:32.896 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5800 IOs, 2262331392 bytes 2024-09-17T23:19:32.979 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5900 IOs, 2266736640 bytes 2024-09-17T23:19:33.082 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6000 IOs, 2291322880 bytes 2024-09-17T23:19:33.206 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6100 IOs, 2335785984 bytes 2024-09-17T23:19:33.325 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6200 IOs, 2378110976 bytes 2024-09-17T23:19:33.407 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6300 IOs, 2378722304 bytes 2024-09-17T23:19:33.521 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6400 IOs, 2412280832 bytes 2024-09-17T23:19:33.617 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6500 IOs, 2424892416 bytes 2024-09-17T23:19:33.777 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6600 IOs, 2508284928 bytes 2024-09-17T23:19:33.895 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6700 IOs, 2548769792 bytes 2024-09-17T23:19:34.007 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6800 IOs, 2578522112 bytes 2024-09-17T23:19:34.145 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6900 IOs, 2635886592 bytes 2024-09-17T23:19:34.226 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7000 IOs, 2636499968 bytes 2024-09-17T23:19:34.306 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7100 IOs, 2637117440 bytes 2024-09-17T23:19:34.474 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7200 IOs, 2729650176 bytes 2024-09-17T23:19:34.601 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7300 IOs, 2782424064 bytes 2024-09-17T23:19:34.685 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7400 IOs, 2788555776 bytes 2024-09-17T23:19:34.828 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7500 IOs, 2848280576 bytes 2024-09-17T23:19:34.961 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7600 IOs, 2904382464 bytes 2024-09-17T23:19:35.090 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7700 IOs, 2961801216 bytes 2024-09-17T23:19:35.198 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7800 IOs, 2987528192 bytes 2024-09-17T23:19:35.281 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7900 IOs, 2988143616 bytes 2024-09-17T23:19:35.382 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8000 IOs, 3012781056 bytes 2024-09-17T23:19:35.498 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8100 IOs, 3053127680 bytes 2024-09-17T23:19:35.607 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8200 IOs, 3080782848 bytes 2024-09-17T23:19:35.719 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8300 IOs, 3113935872 bytes 2024-09-17T23:19:35.833 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8400 IOs, 3150163968 bytes 2024-09-17T23:19:35.960 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8500 IOs, 3198602240 bytes 2024-09-17T23:19:36.101 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8600 IOs, 3284066304 bytes 2024-09-17T23:19:36.264 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8700 IOs, 3349483520 bytes 2024-09-17T23:19:36.348 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8800 IOs, 3355160576 bytes 2024-09-17T23:19:36.430 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8900 IOs, 3355808768 bytes 2024-09-17T23:19:36.570 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9000 IOs, 3415454720 bytes 2024-09-17T23:19:36.659 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9100 IOs, 3423964160 bytes 2024-09-17T23:19:36.759 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9200 IOs, 3444691968 bytes 2024-09-17T23:19:36.864 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9300 IOs, 3467610112 bytes 2024-09-17T23:19:37.004 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9400 IOs, 3526209536 bytes 2024-09-17T23:19:37.089 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9500 IOs, 3536180224 bytes 2024-09-17T23:19:37.222 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9600 IOs, 3588064256 bytes 2024-09-17T23:19:37.336 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9700 IOs, 3625994240 bytes 2024-09-17T23:19:37.445 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9800 IOs, 3657997312 bytes 2024-09-17T23:19:37.569 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9900 IOs, 3708608512 bytes 2024-09-17T23:19:37.668 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10000 IOs, 3725238272 bytes 2024-09-17T23:19:37.778 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10100 IOs, 3775233024 bytes 2024-09-17T23:19:37.941 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10200 IOs, 3846199296 bytes 2024-09-17T23:19:38.099 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10300 IOs, 3925477376 bytes 2024-09-17T23:19:38.251 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10400 IOs, 3997796352 bytes 2024-09-17T23:19:38.379 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10500 IOs, 4068756480 bytes 2024-09-17T23:19:38.515 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10600 IOs, 4117784576 bytes 2024-09-17T23:19:38.624 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10700 IOs, 4148841472 bytes 2024-09-17T23:19:38.764 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10800 IOs, 4214787072 bytes 2024-09-17T23:19:38.875 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10900 IOs, 4242457600 bytes 2024-09-17T23:19:38.999 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:11000 IOs, 4287285248 bytes 2024-09-17T23:19:39.143 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:11100 IOs, 4354394112 bytes 2024-09-17T23:19:39.238 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:11200 IOs, 4367493120 bytes 2024-09-17T23:19:39.339 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:11300 IOs, 4385453056 bytes 2024-09-17T23:19:39.433 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:11400 IOs, 4399868928 bytes 2024-09-17T23:19:39.578 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:11500 IOs, 4473790464 bytes 2024-09-17T23:19:39.696 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:11600 IOs, 4499398656 bytes 2024-09-17T23:19:39.854 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:11700 IOs, 4581560320 bytes 2024-09-17T23:19:40.012 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:11800 IOs, 4671852544 bytes 2024-09-17T23:19:40.074 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ error_code=124 2024-09-17T23:19:40.074 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set -e 2024-09-17T23:19:40.074 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' 124 -eq 124 ']' 2024-09-17T23:19:40.074 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:19:40.074 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:19:40.074 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_20 2024-09-17T23:19:40.074 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_image_and_enable_mirror cluster2 mirror image_20 snapshot 128M 2024-09-17T23:19:40.074 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:19:40.075 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:19:40.075 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:19:40.076 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:19:40.077 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_20 2024-09-17T23:19:40.077 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:19:40.077 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=snapshot 2024-09-17T23:19:40.077 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n snapshot ']' 2024-09-17T23:19:40.077 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:19:40.077 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_image cluster2 mirror image_20 128M 2024-09-17T23:19:40.077 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:19:40.077 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:19:40.077 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:19:40.077 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:19:40.077 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_20 2024-09-17T23:19:40.077 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:19:40.077 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local size=128 2024-09-17T23:19:40.078 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n 128M ']' 2024-09-17T23:19:40.078 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ size=128M 2024-09-17T23:19:40.078 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:19:40.078 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_20 2024-09-17T23:19:40.128 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ is_pool_mirror_mode_image mirror 2024-09-17T23:19:40.129 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:19:40.129 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' image = image ']' 2024-09-17T23:19:40.129 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:19:40.129 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ enable_mirror cluster2 mirror image_20 snapshot 2024-09-17T23:19:40.129 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:19:40.129 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:19:40.129 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_20 2024-09-17T23:19:40.129 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=snapshot 2024-09-17T23:19:40.129 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_20 snapshot 2024-09-17T23:19:41.022 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Mirroring enabled 2024-09-17T23:19:41.034 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 info mirror/image_20 2024-09-17T23:19:41.075 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'image_20': 2024-09-17T23:19:41.075 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 128 MiB in 32 objects 2024-09-17T23:19:41.075 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:19:41.075 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 1 2024-09-17T23:19:41.075 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 136c867b15da 2024-09-17T23:19:41.075 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.136c867b15da 2024-09-17T23:19:41.075 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:19:41.075 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-17T23:19:41.076 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:19:41.076 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:19:41.076 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:19:40 2024 2024-09-17T23:19:41.076 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:19:40 2024 2024-09-17T23:19:41.076 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:19:40 2024 2024-09-17T23:19:41.076 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:19:41.076 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:19:41.076 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: 04cb45cc-a535-47bd-b9a9-46bf5ca4fe67 2024-09-17T23:19:41.076 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: true 2024-09-17T23:19:41.078 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n '' ']' 2024-09-17T23:19:41.079 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ stress_write_image cluster2 mirror image_20 2024-09-17T23:19:41.079 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:19:41.079 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:19:41.079 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_20 2024-09-17T23:19:41.079 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-17T23:19:41.080 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local duration=12 2024-09-17T23:19:41.081 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set +e 2024-09-17T23:19:41.081 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_20 --debug-rbd=20 --debug-journaler=20 2024-09-17T23:19:41.276 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:100 IOs, 25630720 bytes 2024-09-17T23:19:41.389 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:200 IOs, 62263296 bytes 2024-09-17T23:19:41.573 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:300 IOs, 175153152 bytes 2024-09-17T23:19:41.724 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:400 IOs, 250817536 bytes 2024-09-17T23:19:41.857 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:500 IOs, 308019200 bytes 2024-09-17T23:19:41.969 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:600 IOs, 341426176 bytes 2024-09-17T23:19:42.079 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:700 IOs, 393497600 bytes 2024-09-17T23:19:42.248 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:800 IOs, 471846912 bytes 2024-09-17T23:19:42.367 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:900 IOs, 522704896 bytes 2024-09-17T23:19:42.508 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1000 IOs, 596114432 bytes 2024-09-17T23:19:42.625 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1100 IOs, 618439680 bytes 2024-09-17T23:19:42.747 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1200 IOs, 662066176 bytes 2024-09-17T23:19:42.817 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.46831556282198894) > weight (0.25) 2024-09-17T23:19:42.817 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2024-09-17T23:19:42.817 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2024-09-17T23:19:42.818 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2024-09-17T23:19:42.820 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi038.stderr:2024-09-17T23:19:42.817+0000 7f5a4756f640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 37386) UID: 1000 2024-09-17T23:19:42.841 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1300 IOs, 683805696 bytes 2024-09-17T23:19:42.972 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1400 IOs, 725835776 bytes 2024-09-17T23:19:43.087 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1500 IOs, 760330240 bytes 2024-09-17T23:19:43.220 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1600 IOs, 815693824 bytes 2024-09-17T23:19:43.351 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1700 IOs, 874777600 bytes 2024-09-17T23:19:43.458 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1800 IOs, 901349376 bytes 2024-09-17T23:19:43.612 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1900 IOs, 980712448 bytes 2024-09-17T23:19:43.715 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2000 IOs, 1004912640 bytes 2024-09-17T23:19:43.852 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2100 IOs, 1062947840 bytes 2024-09-17T23:19:43.964 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2200 IOs, 1095958528 bytes 2024-09-17T23:19:44.056 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2300 IOs, 1106132992 bytes 2024-09-17T23:19:44.176 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2400 IOs, 1150068736 bytes 2024-09-17T23:19:44.288 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2500 IOs, 1186344960 bytes 2024-09-17T23:19:44.417 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2600 IOs, 1242775552 bytes 2024-09-17T23:19:44.521 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2700 IOs, 1265155072 bytes 2024-09-17T23:19:44.650 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2800 IOs, 1325040640 bytes 2024-09-17T23:19:44.747 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2900 IOs, 1337816064 bytes 2024-09-17T23:19:44.872 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3000 IOs, 1379140608 bytes 2024-09-17T23:19:45.029 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3100 IOs, 1456107520 bytes 2024-09-17T23:19:45.132 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3200 IOs, 1498883072 bytes 2024-09-17T23:19:45.265 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3300 IOs, 1541330944 bytes 2024-09-17T23:19:45.370 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3400 IOs, 1568343040 bytes 2024-09-17T23:19:45.485 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3500 IOs, 1608792064 bytes 2024-09-17T23:19:45.588 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3600 IOs, 1632220160 bytes 2024-09-17T23:19:45.691 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3700 IOs, 1666284544 bytes 2024-09-17T23:19:45.839 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3800 IOs, 1727728640 bytes 2024-09-17T23:19:45.923 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3900 IOs, 1728347136 bytes 2024-09-17T23:19:46.032 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4000 IOs, 1755426816 bytes 2024-09-17T23:19:46.115 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4100 IOs, 1756038144 bytes 2024-09-17T23:19:46.258 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4200 IOs, 1823384576 bytes 2024-09-17T23:19:46.364 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4300 IOs, 1854118912 bytes 2024-09-17T23:19:46.516 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4400 IOs, 1932831744 bytes 2024-09-17T23:19:46.683 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4500 IOs, 2020390912 bytes 2024-09-17T23:19:46.758 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4600 IOs, 2021018624 bytes 2024-09-17T23:19:46.872 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4700 IOs, 2057955328 bytes 2024-09-17T23:19:47.009 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4800 IOs, 2119204864 bytes 2024-09-17T23:19:47.130 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4900 IOs, 2159406080 bytes 2024-09-17T23:19:47.236 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5000 IOs, 2193366016 bytes 2024-09-17T23:19:47.404 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5100 IOs, 2277258240 bytes 2024-09-17T23:19:47.527 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5200 IOs, 2330251264 bytes 2024-09-17T23:19:47.633 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5300 IOs, 2347813888 bytes 2024-09-17T23:19:47.745 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5400 IOs, 2381626368 bytes 2024-09-17T23:19:47.829 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5500 IOs, 2382236672 bytes 2024-09-17T23:19:47.973 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5600 IOs, 2459939840 bytes 2024-09-17T23:19:48.067 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5700 IOs, 2460551168 bytes 2024-09-17T23:19:48.159 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5800 IOs, 2486403072 bytes 2024-09-17T23:19:48.327 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5900 IOs, 2563170304 bytes 2024-09-17T23:19:48.424 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6000 IOs, 2580487168 bytes 2024-09-17T23:19:48.545 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6100 IOs, 2630141952 bytes 2024-09-17T23:19:48.717 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6200 IOs, 2727211008 bytes 2024-09-17T23:19:48.826 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6300 IOs, 2760266752 bytes 2024-09-17T23:19:48.971 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6400 IOs, 2835020800 bytes 2024-09-17T23:19:49.074 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6500 IOs, 2848649216 bytes 2024-09-17T23:19:49.179 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6600 IOs, 2873236480 bytes 2024-09-17T23:19:49.304 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6700 IOs, 2923993088 bytes 2024-09-17T23:19:49.442 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6800 IOs, 2982150144 bytes 2024-09-17T23:19:49.555 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6900 IOs, 3019832320 bytes 2024-09-17T23:19:49.677 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7000 IOs, 3062254592 bytes 2024-09-17T23:19:49.770 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7100 IOs, 3078829056 bytes 2024-09-17T23:19:49.905 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7200 IOs, 3132963840 bytes 2024-09-17T23:19:49.986 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7300 IOs, 3136672768 bytes 2024-09-17T23:19:50.087 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7400 IOs, 3152709632 bytes 2024-09-17T23:19:50.230 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7500 IOs, 3218407424 bytes 2024-09-17T23:19:50.348 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7600 IOs, 3256953856 bytes 2024-09-17T23:19:50.455 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7700 IOs, 3289127936 bytes 2024-09-17T23:19:50.612 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7800 IOs, 3366960128 bytes 2024-09-17T23:19:50.757 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7900 IOs, 3438419968 bytes 2024-09-17T23:19:50.819 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2024-09-17T23:19:50.819 INFO:teuthology.orchestra.run:waiting for 600 2024-09-17T23:19:50.819 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2024-09-17T23:19:50.819 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2024-09-17T23:19:50.819 DEBUG:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2024-09-17T23:19:50.823 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2024-09-17T23:19:50.823 INFO:tasks.rbd_mirror_thrash:waiting for 73 secs before thrashing 2024-09-17T23:19:50.867 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8000 IOs, 3463374848 bytes 2024-09-17T23:19:50.983 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8100 IOs, 3487388672 bytes 2024-09-17T23:19:51.159 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8200 IOs, 3567819776 bytes 2024-09-17T23:19:51.289 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8300 IOs, 3618183168 bytes 2024-09-17T23:19:51.456 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8400 IOs, 3707865088 bytes 2024-09-17T23:19:51.534 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8500 IOs, 3708498944 bytes 2024-09-17T23:19:51.687 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8600 IOs, 3786707968 bytes 2024-09-17T23:19:51.783 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8700 IOs, 3805167616 bytes 2024-09-17T23:19:51.874 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8800 IOs, 3814222848 bytes 2024-09-17T23:19:51.963 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8900 IOs, 3826092032 bytes 2024-09-17T23:19:52.115 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9000 IOs, 3901642752 bytes 2024-09-17T23:19:52.242 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9100 IOs, 3938196480 bytes 2024-09-17T23:19:52.367 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9200 IOs, 3989763072 bytes 2024-09-17T23:19:52.485 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9300 IOs, 4027114496 bytes 2024-09-17T23:19:52.628 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9400 IOs, 4091694080 bytes 2024-09-17T23:19:52.734 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9500 IOs, 4123364352 bytes 2024-09-17T23:19:52.866 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9600 IOs, 4173197312 bytes 2024-09-17T23:19:53.007 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9700 IOs, 4232952832 bytes 2024-09-17T23:19:53.087 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ error_code=124 2024-09-17T23:19:53.087 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set -e 2024-09-17T23:19:53.087 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' 124 -eq 124 ']' 2024-09-17T23:19:53.087 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:19:53.087 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:19:53.087 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_21 2024-09-17T23:19:53.087 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_image_and_enable_mirror cluster2 mirror image_21 snapshot 128M 2024-09-17T23:19:53.088 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:19:53.088 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:19:53.088 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:19:53.088 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:19:53.088 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_21 2024-09-17T23:19:53.088 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:19:53.088 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=snapshot 2024-09-17T23:19:53.088 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n snapshot ']' 2024-09-17T23:19:53.088 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:19:53.088 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_image cluster2 mirror image_21 128M 2024-09-17T23:19:53.088 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:19:53.088 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:19:53.089 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:19:53.089 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:19:53.089 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_21 2024-09-17T23:19:53.089 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:19:53.089 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local size=128 2024-09-17T23:19:53.090 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n 128M ']' 2024-09-17T23:19:53.090 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ size=128M 2024-09-17T23:19:53.090 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:19:53.090 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_21 2024-09-17T23:19:53.137 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ is_pool_mirror_mode_image mirror 2024-09-17T23:19:53.137 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:19:53.137 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' image = image ']' 2024-09-17T23:19:53.137 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:19:53.137 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ enable_mirror cluster2 mirror image_21 snapshot 2024-09-17T23:19:53.137 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:19:53.137 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:19:53.138 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_21 2024-09-17T23:19:53.138 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=snapshot 2024-09-17T23:19:53.138 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_21 snapshot 2024-09-17T23:19:54.045 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Mirroring enabled 2024-09-17T23:19:54.056 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 info mirror/image_21 2024-09-17T23:19:54.097 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'image_21': 2024-09-17T23:19:54.097 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 128 MiB in 32 objects 2024-09-17T23:19:54.097 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:19:54.097 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 1 2024-09-17T23:19:54.097 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 137d695ecbb7 2024-09-17T23:19:54.098 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.137d695ecbb7 2024-09-17T23:19:54.098 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:19:54.098 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-17T23:19:54.098 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:19:54.098 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:19:54.098 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:19:53 2024 2024-09-17T23:19:54.098 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:19:53 2024 2024-09-17T23:19:54.098 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:19:53 2024 2024-09-17T23:19:54.098 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:19:54.098 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:19:54.098 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: 7f670507-4ecc-426a-a941-5287c614d52d 2024-09-17T23:19:54.098 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: true 2024-09-17T23:19:54.100 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n '' ']' 2024-09-17T23:19:54.100 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ stress_write_image cluster2 mirror image_21 2024-09-17T23:19:54.100 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:19:54.101 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:19:54.101 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_21 2024-09-17T23:19:54.101 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-17T23:19:54.102 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local duration=10 2024-09-17T23:19:54.103 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set +e 2024-09-17T23:19:54.103 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_21 --debug-rbd=20 --debug-journaler=20 2024-09-17T23:19:54.356 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:100 IOs, 98823168 bytes 2024-09-17T23:19:54.488 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:200 IOs, 169754624 bytes 2024-09-17T23:19:54.610 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:300 IOs, 207926272 bytes 2024-09-17T23:19:54.723 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:400 IOs, 252993536 bytes 2024-09-17T23:19:54.816 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:500 IOs, 282334208 bytes 2024-09-17T23:19:54.948 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:600 IOs, 319661056 bytes 2024-09-17T23:19:55.074 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:700 IOs, 380601344 bytes 2024-09-17T23:19:55.254 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:800 IOs, 473550848 bytes 2024-09-17T23:19:55.391 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:900 IOs, 531892224 bytes 2024-09-17T23:19:55.508 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1000 IOs, 566645760 bytes 2024-09-17T23:19:55.638 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1100 IOs, 623245312 bytes 2024-09-17T23:19:55.781 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1200 IOs, 686233600 bytes 2024-09-17T23:19:55.911 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1300 IOs, 732631040 bytes 2024-09-17T23:19:56.010 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1400 IOs, 776181760 bytes 2024-09-17T23:19:56.136 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1500 IOs, 792427520 bytes 2024-09-17T23:19:56.220 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1600 IOs, 796741632 bytes 2024-09-17T23:19:56.350 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1700 IOs, 844431360 bytes 2024-09-17T23:19:56.467 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1800 IOs, 883250176 bytes 2024-09-17T23:19:56.575 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1900 IOs, 918597632 bytes 2024-09-17T23:19:56.662 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2000 IOs, 921903104 bytes 2024-09-17T23:19:56.744 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2100 IOs, 924242944 bytes 2024-09-17T23:19:56.863 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2200 IOs, 963808256 bytes 2024-09-17T23:19:56.992 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2300 IOs, 1005770752 bytes 2024-09-17T23:19:57.138 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2400 IOs, 1076711424 bytes 2024-09-17T23:19:57.263 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2500 IOs, 1108191232 bytes 2024-09-17T23:19:57.343 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2600 IOs, 1108816896 bytes 2024-09-17T23:19:57.440 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2700 IOs, 1127348224 bytes 2024-09-17T23:19:57.534 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2800 IOs, 1143343104 bytes 2024-09-17T23:19:57.610 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2900 IOs, 1143969792 bytes 2024-09-17T23:19:57.716 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3000 IOs, 1171083264 bytes 2024-09-17T23:19:57.820 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3100 IOs, 1196803072 bytes 2024-09-17T23:19:58.004 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3200 IOs, 1295039488 bytes 2024-09-17T23:19:58.122 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3300 IOs, 1327133696 bytes 2024-09-17T23:19:58.252 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3400 IOs, 1380763648 bytes 2024-09-17T23:19:58.372 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3500 IOs, 1421547520 bytes 2024-09-17T23:19:58.466 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3600 IOs, 1434628096 bytes 2024-09-17T23:19:58.582 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3700 IOs, 1476847616 bytes 2024-09-17T23:19:58.707 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3800 IOs, 1541708800 bytes 2024-09-17T23:19:58.828 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3900 IOs, 1567503360 bytes 2024-09-17T23:19:58.921 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4000 IOs, 1583711232 bytes 2024-09-17T23:19:59.040 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4100 IOs, 1618938880 bytes 2024-09-17T23:19:59.125 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4200 IOs, 1622131712 bytes 2024-09-17T23:19:59.256 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4300 IOs, 1676621824 bytes 2024-09-17T23:19:59.398 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4400 IOs, 1754459136 bytes 2024-09-17T23:19:59.534 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4500 IOs, 1803881472 bytes 2024-09-17T23:19:59.625 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4600 IOs, 1815127040 bytes 2024-09-17T23:19:59.713 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4700 IOs, 1824612352 bytes 2024-09-17T23:19:59.892 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4800 IOs, 1938160640 bytes 2024-09-17T23:20:00.057 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4900 IOs, 2017920000 bytes 2024-09-17T23:20:00.148 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5000 IOs, 2025467904 bytes 2024-09-17T23:20:00.263 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5100 IOs, 2062875648 bytes 2024-09-17T23:20:00.342 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5200 IOs, 2063481856 bytes 2024-09-17T23:20:00.423 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5300 IOs, 2064084992 bytes 2024-09-17T23:20:00.544 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5400 IOs, 2105181184 bytes 2024-09-17T23:20:00.625 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5500 IOs, 2109899776 bytes 2024-09-17T23:20:00.708 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5600 IOs, 2114494464 bytes 2024-09-17T23:20:00.832 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5700 IOs, 2162115584 bytes 2024-09-17T23:20:00.934 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5800 IOs, 2185936896 bytes 2024-09-17T23:20:01.023 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5900 IOs, 2194188288 bytes 2024-09-17T23:20:01.187 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6000 IOs, 2281098240 bytes 2024-09-17T23:20:01.278 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6100 IOs, 2289644544 bytes 2024-09-17T23:20:01.418 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6200 IOs, 2363953152 bytes 2024-09-17T23:20:01.515 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6300 IOs, 2370022400 bytes 2024-09-17T23:20:01.659 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6400 IOs, 2436775936 bytes 2024-09-17T23:20:01.749 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6500 IOs, 2443610112 bytes 2024-09-17T23:20:01.839 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6600 IOs, 2453452800 bytes 2024-09-17T23:20:01.985 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6700 IOs, 2523459584 bytes 2024-09-17T23:20:02.038 INFO:tasks.ceph.cluster2.osd.0.smithi038.stderr:2024-09-17T23:20:02.035+0000 7fa26a72e640 -1 reset not still connected to 0x55b309c58dd0 2024-09-17T23:20:02.038 INFO:tasks.ceph.cluster2.osd.0.smithi038.stderr:2024-09-17T23:20:02.035+0000 7fa26a72e640 -1 reset not still connected to 0x55b309c59450 2024-09-17T23:20:02.115 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6800 IOs, 2566147072 bytes 2024-09-17T23:20:02.229 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6900 IOs, 2600708096 bytes 2024-09-17T23:20:02.340 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7000 IOs, 2638001152 bytes 2024-09-17T23:20:02.473 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7100 IOs, 2684427264 bytes 2024-09-17T23:20:02.595 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7200 IOs, 2728806400 bytes 2024-09-17T23:20:02.684 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7300 IOs, 2741754880 bytes 2024-09-17T23:20:02.796 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7400 IOs, 2771197952 bytes 2024-09-17T23:20:02.895 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7500 IOs, 2788679680 bytes 2024-09-17T23:20:02.983 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7600 IOs, 2799506432 bytes 2024-09-17T23:20:03.123 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7700 IOs, 2856875008 bytes 2024-09-17T23:20:03.269 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7800 IOs, 2924659712 bytes 2024-09-17T23:20:03.385 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7900 IOs, 2975601664 bytes 2024-09-17T23:20:03.502 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8000 IOs, 3001400320 bytes 2024-09-17T23:20:03.604 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8100 IOs, 3026449408 bytes 2024-09-17T23:20:03.703 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8200 IOs, 3050446848 bytes 2024-09-17T23:20:03.814 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8300 IOs, 3097049088 bytes 2024-09-17T23:20:03.992 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8400 IOs, 3198118912 bytes 2024-09-17T23:20:04.108 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ error_code=124 2024-09-17T23:20:04.108 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set -e 2024-09-17T23:20:04.108 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' 124 -eq 124 ']' 2024-09-17T23:20:04.109 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:20:04.109 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:20:04.109 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_22 2024-09-17T23:20:04.109 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_image_and_enable_mirror cluster2 mirror image_22 snapshot 128M 2024-09-17T23:20:04.109 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:20:04.109 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:20:04.109 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:20:04.109 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:20:04.109 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_22 2024-09-17T23:20:04.109 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:20:04.109 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=snapshot 2024-09-17T23:20:04.109 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n snapshot ']' 2024-09-17T23:20:04.109 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:20:04.109 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_image cluster2 mirror image_22 128M 2024-09-17T23:20:04.110 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:20:04.110 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:20:04.110 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:20:04.110 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:20:04.110 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_22 2024-09-17T23:20:04.110 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:20:04.110 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local size=128 2024-09-17T23:20:04.110 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n 128M ']' 2024-09-17T23:20:04.111 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ size=128M 2024-09-17T23:20:04.111 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:20:04.111 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_22 2024-09-17T23:20:04.158 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ is_pool_mirror_mode_image mirror 2024-09-17T23:20:04.159 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:20:04.159 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' image = image ']' 2024-09-17T23:20:04.159 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:20:04.159 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ enable_mirror cluster2 mirror image_22 snapshot 2024-09-17T23:20:04.159 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:20:04.159 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:20:04.159 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_22 2024-09-17T23:20:04.159 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=snapshot 2024-09-17T23:20:04.159 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_22 snapshot 2024-09-17T23:20:05.066 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Mirroring enabled 2024-09-17T23:20:05.081 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 info mirror/image_22 2024-09-17T23:20:05.121 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'image_22': 2024-09-17T23:20:05.121 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 128 MiB in 32 objects 2024-09-17T23:20:05.121 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:20:05.121 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 1 2024-09-17T23:20:05.121 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 13896332c9f4 2024-09-17T23:20:05.121 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.13896332c9f4 2024-09-17T23:20:05.121 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:20:05.121 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-17T23:20:05.121 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:20:05.121 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:20:05.121 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:20:04 2024 2024-09-17T23:20:05.121 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:20:04 2024 2024-09-17T23:20:05.121 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:20:04 2024 2024-09-17T23:20:05.122 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:20:05.122 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:20:05.122 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: 1068bb88-a6ef-49c0-a048-fafa200170dc 2024-09-17T23:20:05.122 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: true 2024-09-17T23:20:05.123 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n '' ']' 2024-09-17T23:20:05.123 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ stress_write_image cluster2 mirror image_22 2024-09-17T23:20:05.123 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:20:05.123 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:20:05.123 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_22 2024-09-17T23:20:05.123 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-17T23:20:05.124 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local duration=5 2024-09-17T23:20:05.124 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set +e 2024-09-17T23:20:05.124 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_22 --debug-rbd=20 --debug-journaler=20 2024-09-17T23:20:05.321 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:100 IOs, 32787456 bytes 2024-09-17T23:20:05.471 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:200 IOs, 108901376 bytes 2024-09-17T23:20:05.559 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:300 IOs, 120504320 bytes 2024-09-17T23:20:05.660 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:400 IOs, 145324032 bytes 2024-09-17T23:20:05.774 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:500 IOs, 197545984 bytes 2024-09-17T23:20:05.940 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:600 IOs, 279553024 bytes 2024-09-17T23:20:06.111 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:700 IOs, 373539840 bytes 2024-09-17T23:20:06.230 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:800 IOs, 411005952 bytes 2024-09-17T23:20:06.348 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:900 IOs, 454069248 bytes 2024-09-17T23:20:06.465 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1000 IOs, 491985920 bytes 2024-09-17T23:20:06.599 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1100 IOs, 562273280 bytes 2024-09-17T23:20:06.746 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1200 IOs, 624788480 bytes 2024-09-17T23:20:06.859 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1300 IOs, 658403328 bytes 2024-09-17T23:20:06.969 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1400 IOs, 691811328 bytes 2024-09-17T23:20:07.113 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1500 IOs, 750935040 bytes 2024-09-17T23:20:07.219 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1600 IOs, 773444608 bytes 2024-09-17T23:20:07.320 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1700 IOs, 824624128 bytes 2024-09-17T23:20:07.447 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1800 IOs, 839148544 bytes 2024-09-17T23:20:07.531 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1900 IOs, 852724736 bytes 2024-09-17T23:20:07.680 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2000 IOs, 934052864 bytes 2024-09-17T23:20:07.782 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2100 IOs, 945384448 bytes 2024-09-17T23:20:07.901 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2200 IOs, 990757888 bytes 2024-09-17T23:20:08.036 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2300 IOs, 1050396672 bytes 2024-09-17T23:20:08.132 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2400 IOs, 1069872128 bytes 2024-09-17T23:20:08.258 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2500 IOs, 1116001280 bytes 2024-09-17T23:20:08.373 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2600 IOs, 1149304832 bytes 2024-09-17T23:20:08.470 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2700 IOs, 1163356160 bytes 2024-09-17T23:20:08.600 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2800 IOs, 1218223104 bytes 2024-09-17T23:20:08.744 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2900 IOs, 1285962752 bytes 2024-09-17T23:20:08.836 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3000 IOs, 1293083648 bytes 2024-09-17T23:20:08.993 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3100 IOs, 1377518592 bytes 2024-09-17T23:20:09.090 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3200 IOs, 1396438016 bytes 2024-09-17T23:20:09.234 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3300 IOs, 1459263488 bytes 2024-09-17T23:20:09.339 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3400 IOs, 1480540160 bytes 2024-09-17T23:20:09.443 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3500 IOs, 1504800768 bytes 2024-09-17T23:20:09.542 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3600 IOs, 1524712448 bytes 2024-09-17T23:20:09.650 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3700 IOs, 1566434304 bytes 2024-09-17T23:20:09.789 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3800 IOs, 1616506880 bytes 2024-09-17T23:20:09.919 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3900 IOs, 1674547200 bytes 2024-09-17T23:20:10.081 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4000 IOs, 1753767936 bytes 2024-09-17T23:20:10.130 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ error_code=124 2024-09-17T23:20:10.130 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set -e 2024-09-17T23:20:10.130 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' 124 -eq 124 ']' 2024-09-17T23:20:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:20:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:20:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_23 2024-09-17T23:20:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_image_and_enable_mirror cluster2 mirror image_23 snapshot 128M 2024-09-17T23:20:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:20:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:20:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:20:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:20:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_23 2024-09-17T23:20:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:20:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=snapshot 2024-09-17T23:20:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n snapshot ']' 2024-09-17T23:20:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:20:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_image cluster2 mirror image_23 128M 2024-09-17T23:20:10.132 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:20:10.132 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:20:10.133 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:20:10.133 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:20:10.133 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_23 2024-09-17T23:20:10.133 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:20:10.133 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local size=128 2024-09-17T23:20:10.133 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n 128M ']' 2024-09-17T23:20:10.133 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ size=128M 2024-09-17T23:20:10.133 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:20:10.133 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_23 2024-09-17T23:20:10.180 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ is_pool_mirror_mode_image mirror 2024-09-17T23:20:10.180 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:20:10.180 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' image = image ']' 2024-09-17T23:20:10.181 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:20:10.181 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ enable_mirror cluster2 mirror image_23 snapshot 2024-09-17T23:20:10.181 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:20:10.181 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:20:10.181 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_23 2024-09-17T23:20:10.181 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=snapshot 2024-09-17T23:20:10.181 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_23 snapshot 2024-09-17T23:20:11.078 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Mirroring enabled 2024-09-17T23:20:11.090 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 info mirror/image_23 2024-09-17T23:20:11.132 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'image_23': 2024-09-17T23:20:11.132 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 128 MiB in 32 objects 2024-09-17T23:20:11.132 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:20:11.132 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 1 2024-09-17T23:20:11.132 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 139513c4bf37 2024-09-17T23:20:11.133 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.139513c4bf37 2024-09-17T23:20:11.133 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:20:11.133 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-17T23:20:11.133 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:20:11.133 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:20:11.133 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:20:10 2024 2024-09-17T23:20:11.133 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:20:10 2024 2024-09-17T23:20:11.133 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:20:10 2024 2024-09-17T23:20:11.133 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:20:11.133 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:20:11.133 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: 7b178615-290c-4f56-9926-be968e7bf83a 2024-09-17T23:20:11.134 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: true 2024-09-17T23:20:11.135 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n '' ']' 2024-09-17T23:20:11.135 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ stress_write_image cluster2 mirror image_23 2024-09-17T23:20:11.136 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:20:11.136 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:20:11.136 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_23 2024-09-17T23:20:11.136 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-17T23:20:11.138 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local duration=13 2024-09-17T23:20:11.138 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set +e 2024-09-17T23:20:11.138 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_23 --debug-rbd=20 --debug-journaler=20 2024-09-17T23:20:11.333 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:100 IOs, 40249344 bytes 2024-09-17T23:20:11.438 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:200 IOs, 55202816 bytes 2024-09-17T23:20:11.552 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:300 IOs, 94961664 bytes 2024-09-17T23:20:11.650 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:400 IOs, 118004736 bytes 2024-09-17T23:20:11.775 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:500 IOs, 168869888 bytes 2024-09-17T23:20:11.936 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:600 IOs, 251763712 bytes 2024-09-17T23:20:12.025 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:700 IOs, 265397248 bytes 2024-09-17T23:20:12.155 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:800 IOs, 317340672 bytes 2024-09-17T23:20:12.276 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:900 IOs, 356327424 bytes 2024-09-17T23:20:12.386 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1000 IOs, 387072000 bytes 2024-09-17T23:20:12.534 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1100 IOs, 457065472 bytes 2024-09-17T23:20:12.617 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1200 IOs, 467825664 bytes 2024-09-17T23:20:12.709 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1300 IOs, 477190144 bytes 2024-09-17T23:20:12.813 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1400 IOs, 500029440 bytes 2024-09-17T23:20:12.943 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1500 IOs, 551769088 bytes 2024-09-17T23:20:13.043 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1600 IOs, 575469568 bytes 2024-09-17T23:20:13.170 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1700 IOs, 613895168 bytes 2024-09-17T23:20:13.307 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1800 IOs, 667786240 bytes 2024-09-17T23:20:13.434 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1900 IOs, 720854016 bytes 2024-09-17T23:20:13.636 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2000 IOs, 766867456 bytes 2024-09-17T23:20:13.777 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2100 IOs, 852513792 bytes 2024-09-17T23:20:13.932 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2200 IOs, 908534784 bytes 2024-09-17T23:20:14.050 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2300 IOs, 950686720 bytes 2024-09-17T23:20:14.196 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2400 IOs, 1040575488 bytes 2024-09-17T23:20:14.331 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2500 IOs, 1080908800 bytes 2024-09-17T23:20:14.437 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2600 IOs, 1109409792 bytes 2024-09-17T23:20:14.566 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2700 IOs, 1157916672 bytes 2024-09-17T23:20:14.655 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2800 IOs, 1192048640 bytes 2024-09-17T23:20:14.838 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2900 IOs, 1282796544 bytes 2024-09-17T23:20:14.938 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3000 IOs, 1308276736 bytes 2024-09-17T23:20:15.049 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3100 IOs, 1342897152 bytes 2024-09-17T23:20:15.137 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3200 IOs, 1350277120 bytes 2024-09-17T23:20:15.243 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3300 IOs, 1374934016 bytes 2024-09-17T23:20:15.381 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3400 IOs, 1438375936 bytes 2024-09-17T23:20:15.547 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3500 IOs, 1528968192 bytes 2024-09-17T23:20:15.673 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3600 IOs, 1582508032 bytes 2024-09-17T23:20:15.780 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3700 IOs, 1610458112 bytes 2024-09-17T23:20:15.883 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3800 IOs, 1638659072 bytes 2024-09-17T23:20:16.043 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3900 IOs, 1727237120 bytes 2024-09-17T23:20:16.153 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4000 IOs, 1768145920 bytes 2024-09-17T23:20:16.288 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4100 IOs, 1815969792 bytes 2024-09-17T23:20:16.378 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4200 IOs, 1826948096 bytes 2024-09-17T23:20:16.490 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4300 IOs, 1854053376 bytes 2024-09-17T23:20:16.640 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4400 IOs, 1930076160 bytes 2024-09-17T23:20:16.732 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4500 IOs, 1943571456 bytes 2024-09-17T23:20:16.823 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4600 IOs, 1954513920 bytes 2024-09-17T23:20:16.962 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4700 IOs, 2012380160 bytes 2024-09-17T23:20:17.078 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4800 IOs, 2055472128 bytes 2024-09-17T23:20:17.245 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4900 IOs, 2149075968 bytes 2024-09-17T23:20:17.374 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5000 IOs, 2204218368 bytes 2024-09-17T23:20:17.514 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5100 IOs, 2254988288 bytes 2024-09-17T23:20:17.624 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5200 IOs, 2283286528 bytes 2024-09-17T23:20:17.762 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5300 IOs, 2344663040 bytes 2024-09-17T23:20:17.901 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5400 IOs, 2405192704 bytes 2024-09-17T23:20:18.050 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5500 IOs, 2487303168 bytes 2024-09-17T23:20:18.143 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5600 IOs, 2499547136 bytes 2024-09-17T23:20:18.263 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5700 IOs, 2538318848 bytes 2024-09-17T23:20:18.385 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5800 IOs, 2576458752 bytes 2024-09-17T23:20:18.478 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5900 IOs, 2585461760 bytes 2024-09-17T23:20:18.621 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6000 IOs, 2651649024 bytes 2024-09-17T23:20:18.705 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6100 IOs, 2686436352 bytes 2024-09-17T23:20:18.855 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6200 IOs, 2732964864 bytes 2024-09-17T23:20:18.981 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6300 IOs, 2782605312 bytes 2024-09-17T23:20:19.105 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6400 IOs, 2834234368 bytes 2024-09-17T23:20:19.279 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6500 IOs, 2949486592 bytes 2024-09-17T23:20:19.433 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6600 IOs, 3000323072 bytes 2024-09-17T23:20:19.518 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6700 IOs, 3009448960 bytes 2024-09-17T23:20:19.649 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6800 IOs, 3064209408 bytes 2024-09-17T23:20:19.755 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6900 IOs, 3091188736 bytes 2024-09-17T23:20:19.838 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7000 IOs, 3095505920 bytes 2024-09-17T23:20:19.942 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7100 IOs, 3122833408 bytes 2024-09-17T23:20:20.031 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7200 IOs, 3133079552 bytes 2024-09-17T23:20:20.126 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7300 IOs, 3155771392 bytes 2024-09-17T23:20:20.215 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7400 IOs, 3160344576 bytes 2024-09-17T23:20:20.336 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7500 IOs, 3203714048 bytes 2024-09-17T23:20:20.448 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7600 IOs, 3235351552 bytes 2024-09-17T23:20:20.608 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7700 IOs, 3315134464 bytes 2024-09-17T23:20:20.732 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7800 IOs, 3357616128 bytes 2024-09-17T23:20:20.854 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7900 IOs, 3408335872 bytes 2024-09-17T23:20:20.989 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8000 IOs, 3457875968 bytes 2024-09-17T23:20:21.146 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8100 IOs, 3537501184 bytes 2024-09-17T23:20:21.281 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8200 IOs, 3598097408 bytes 2024-09-17T23:20:21.406 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8300 IOs, 3663425536 bytes 2024-09-17T23:20:21.566 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8400 IOs, 3723686912 bytes 2024-09-17T23:20:21.694 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8500 IOs, 3769121792 bytes 2024-09-17T23:20:21.798 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8600 IOs, 3798914048 bytes 2024-09-17T23:20:21.885 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8700 IOs, 3804961792 bytes 2024-09-17T23:20:21.975 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8800 IOs, 3817844736 bytes 2024-09-17T23:20:22.098 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8900 IOs, 3861823488 bytes 2024-09-17T23:20:22.218 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9000 IOs, 3915580416 bytes 2024-09-17T23:20:22.331 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9100 IOs, 3937960960 bytes 2024-09-17T23:20:22.437 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9200 IOs, 3962008576 bytes 2024-09-17T23:20:22.570 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9300 IOs, 4015340544 bytes 2024-09-17T23:20:22.737 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9400 IOs, 4104640512 bytes 2024-09-17T23:20:22.845 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9500 IOs, 4136775680 bytes 2024-09-17T23:20:22.958 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9600 IOs, 4173298688 bytes 2024-09-17T23:20:23.068 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9700 IOs, 4202055680 bytes 2024-09-17T23:20:23.189 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9800 IOs, 4244885504 bytes 2024-09-17T23:20:23.358 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9900 IOs, 4340704256 bytes 2024-09-17T23:20:23.472 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10000 IOs, 4375796736 bytes 2024-09-17T23:20:23.623 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10100 IOs, 4449345536 bytes 2024-09-17T23:20:23.737 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10200 IOs, 4482816000 bytes 2024-09-17T23:20:23.821 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10300 IOs, 4492862464 bytes 2024-09-17T23:20:23.961 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10400 IOs, 4555545600 bytes 2024-09-17T23:20:24.066 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10500 IOs, 4581821440 bytes 2024-09-17T23:20:24.143 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ error_code=124 2024-09-17T23:20:24.143 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set -e 2024-09-17T23:20:24.143 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' 124 -eq 124 ']' 2024-09-17T23:20:24.143 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:20:24.143 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:20:24.143 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_24 2024-09-17T23:20:24.143 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_image_and_enable_mirror cluster2 mirror image_24 snapshot 128M 2024-09-17T23:20:24.144 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:20:24.144 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:20:24.144 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:20:24.144 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:20:24.144 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_24 2024-09-17T23:20:24.144 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:20:24.144 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=snapshot 2024-09-17T23:20:24.144 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n snapshot ']' 2024-09-17T23:20:24.144 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:20:24.144 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_image cluster2 mirror image_24 128M 2024-09-17T23:20:24.144 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:20:24.145 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:20:24.145 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:20:24.145 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:20:24.145 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_24 2024-09-17T23:20:24.145 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:20:24.145 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local size=128 2024-09-17T23:20:24.145 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n 128M ']' 2024-09-17T23:20:24.145 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ size=128M 2024-09-17T23:20:24.145 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:20:24.146 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_24 2024-09-17T23:20:24.194 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ is_pool_mirror_mode_image mirror 2024-09-17T23:20:24.194 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:20:24.195 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' image = image ']' 2024-09-17T23:20:24.195 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:20:24.195 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ enable_mirror cluster2 mirror image_24 snapshot 2024-09-17T23:20:24.195 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:20:24.195 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:20:24.195 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_24 2024-09-17T23:20:24.195 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=snapshot 2024-09-17T23:20:24.195 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_24 snapshot 2024-09-17T23:20:25.095 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Mirroring enabled 2024-09-17T23:20:25.107 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 info mirror/image_24 2024-09-17T23:20:25.148 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'image_24': 2024-09-17T23:20:25.148 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 128 MiB in 32 objects 2024-09-17T23:20:25.148 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:20:25.148 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 1 2024-09-17T23:20:25.148 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 13a1796c760f 2024-09-17T23:20:25.148 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.13a1796c760f 2024-09-17T23:20:25.148 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:20:25.148 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-17T23:20:25.148 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:20:25.149 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:20:25.149 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:20:24 2024 2024-09-17T23:20:25.149 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:20:24 2024 2024-09-17T23:20:25.149 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:20:24 2024 2024-09-17T23:20:25.149 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:20:25.149 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:20:25.149 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: de674b7e-029c-4272-9b26-d163572b5aea 2024-09-17T23:20:25.149 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: true 2024-09-17T23:20:25.151 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n '' ']' 2024-09-17T23:20:25.151 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ stress_write_image cluster2 mirror image_24 2024-09-17T23:20:25.151 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:20:25.152 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:20:25.152 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_24 2024-09-17T23:20:25.152 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-17T23:20:25.153 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local duration=6 2024-09-17T23:20:25.153 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set +e 2024-09-17T23:20:25.153 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_24 --debug-rbd=20 --debug-journaler=20 2024-09-17T23:20:25.400 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:100 IOs, 83481600 bytes 2024-09-17T23:20:25.566 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:200 IOs, 165384192 bytes 2024-09-17T23:20:25.654 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:300 IOs, 182797312 bytes 2024-09-17T23:20:25.764 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:400 IOs, 215793664 bytes 2024-09-17T23:20:25.893 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:500 IOs, 267137024 bytes 2024-09-17T23:20:25.928 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.4091200555515455) > weight (0.25) 2024-09-17T23:20:25.928 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.45550637031935626) > weight (0.25) 2024-09-17T23:20:25.928 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2024-09-17T23:20:25.928 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2024-09-17T23:20:25.928 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2024-09-17T23:20:25.929 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi038.stderr:2024-09-17T23:20:25.927+0000 7faa0a4b5640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 46714) UID: 1000 2024-09-17T23:20:26.126 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:600 IOs, 418502656 bytes 2024-09-17T23:20:26.233 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:700 IOs, 448343040 bytes 2024-09-17T23:20:26.324 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:800 IOs, 465049600 bytes 2024-09-17T23:20:26.524 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:900 IOs, 585885696 bytes 2024-09-17T23:20:26.664 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1000 IOs, 652495872 bytes 2024-09-17T23:20:26.795 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1100 IOs, 706838528 bytes 2024-09-17T23:20:26.901 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1200 IOs, 734275584 bytes 2024-09-17T23:20:27.039 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1300 IOs, 809961472 bytes 2024-09-17T23:20:27.225 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1400 IOs, 899883008 bytes 2024-09-17T23:20:27.356 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1500 IOs, 950712320 bytes 2024-09-17T23:20:27.483 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1600 IOs, 988724224 bytes 2024-09-17T23:20:27.608 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1700 IOs, 1036955648 bytes 2024-09-17T23:20:27.754 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1800 IOs, 1105238016 bytes 2024-09-17T23:20:27.869 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1900 IOs, 1144544256 bytes 2024-09-17T23:20:27.987 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2000 IOs, 1184380928 bytes 2024-09-17T23:20:28.111 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2100 IOs, 1233174528 bytes 2024-09-17T23:20:28.301 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2200 IOs, 1368756224 bytes 2024-09-17T23:20:28.439 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2300 IOs, 1422658560 bytes 2024-09-17T23:20:28.585 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2400 IOs, 1478602752 bytes 2024-09-17T23:20:28.701 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2500 IOs, 1514932224 bytes 2024-09-17T23:20:28.846 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2600 IOs, 1585087488 bytes 2024-09-17T23:20:28.946 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2700 IOs, 1604335616 bytes 2024-09-17T23:20:29.061 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2800 IOs, 1641919488 bytes 2024-09-17T23:20:29.179 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2900 IOs, 1685027840 bytes 2024-09-17T23:20:29.278 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3000 IOs, 1706986496 bytes 2024-09-17T23:20:29.413 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3100 IOs, 1766117376 bytes 2024-09-17T23:20:29.509 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3200 IOs, 1777437696 bytes 2024-09-17T23:20:29.648 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3300 IOs, 1835385856 bytes 2024-09-17T23:20:29.763 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3400 IOs, 1870284800 bytes 2024-09-17T23:20:29.888 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3500 IOs, 1918358528 bytes 2024-09-17T23:20:29.988 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3600 IOs, 1942060032 bytes 2024-09-17T23:20:30.105 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3700 IOs, 1980445696 bytes 2024-09-17T23:20:30.206 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3800 IOs, 2000015360 bytes 2024-09-17T23:20:30.345 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3900 IOs, 2062990336 bytes 2024-09-17T23:20:30.451 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4000 IOs, 2092013568 bytes 2024-09-17T23:20:30.548 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4100 IOs, 2108418048 bytes 2024-09-17T23:20:30.654 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4200 IOs, 2154342400 bytes 2024-09-17T23:20:30.765 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4300 IOs, 2167060480 bytes 2024-09-17T23:20:30.886 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4400 IOs, 2212133888 bytes 2024-09-17T23:20:31.083 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4500 IOs, 2340270080 bytes 2024-09-17T23:20:31.161 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ error_code=124 2024-09-17T23:20:31.161 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set -e 2024-09-17T23:20:31.161 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' 124 -eq 124 ']' 2024-09-17T23:20:31.161 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:20:31.161 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:20:31.162 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_25 2024-09-17T23:20:31.162 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_image_and_enable_mirror cluster2 mirror image_25 snapshot 128M 2024-09-17T23:20:31.162 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:20:31.162 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:20:31.162 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:20:31.162 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:20:31.162 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_25 2024-09-17T23:20:31.162 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:20:31.162 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=snapshot 2024-09-17T23:20:31.162 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n snapshot ']' 2024-09-17T23:20:31.162 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:20:31.162 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_image cluster2 mirror image_25 128M 2024-09-17T23:20:31.162 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:20:31.162 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:20:31.163 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:20:31.163 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:20:31.163 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_25 2024-09-17T23:20:31.163 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:20:31.163 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local size=128 2024-09-17T23:20:31.163 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n 128M ']' 2024-09-17T23:20:31.163 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ size=128M 2024-09-17T23:20:31.164 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:20:31.164 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_25 2024-09-17T23:20:31.211 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ is_pool_mirror_mode_image mirror 2024-09-17T23:20:31.211 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:20:31.211 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' image = image ']' 2024-09-17T23:20:31.211 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:20:31.211 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ enable_mirror cluster2 mirror image_25 snapshot 2024-09-17T23:20:31.211 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:20:31.211 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:20:31.212 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_25 2024-09-17T23:20:31.212 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=snapshot 2024-09-17T23:20:31.212 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_25 snapshot 2024-09-17T23:20:31.929 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2024-09-17T23:20:31.929 INFO:teuthology.orchestra.run:waiting for 600 2024-09-17T23:20:31.930 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2024-09-17T23:20:31.930 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2024-09-17T23:20:31.930 DEBUG:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2024-09-17T23:20:31.934 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2024-09-17T23:20:31.934 INFO:tasks.rbd_mirror_thrash:waiting for 82 secs before thrashing 2024-09-17T23:20:32.102 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Mirroring enabled 2024-09-17T23:20:32.116 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 info mirror/image_25 2024-09-17T23:20:32.157 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'image_25': 2024-09-17T23:20:32.157 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 128 MiB in 32 objects 2024-09-17T23:20:32.158 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:20:32.158 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 1 2024-09-17T23:20:32.158 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 13ad9b91fde8 2024-09-17T23:20:32.158 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.13ad9b91fde8 2024-09-17T23:20:32.158 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:20:32.158 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-17T23:20:32.158 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:20:32.158 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:20:32.158 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:20:31 2024 2024-09-17T23:20:32.158 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:20:31 2024 2024-09-17T23:20:32.158 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:20:31 2024 2024-09-17T23:20:32.158 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:20:32.158 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:20:32.159 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: d504d61f-d7b1-44e8-a7fe-4d46bee040b2 2024-09-17T23:20:32.159 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: true 2024-09-17T23:20:32.161 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n '' ']' 2024-09-17T23:20:32.161 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ stress_write_image cluster2 mirror image_25 2024-09-17T23:20:32.161 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:20:32.161 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:20:32.161 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_25 2024-09-17T23:20:32.161 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-17T23:20:32.163 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local duration=7 2024-09-17T23:20:32.163 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set +e 2024-09-17T23:20:32.163 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_25 --debug-rbd=20 --debug-journaler=20 2024-09-17T23:20:32.361 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:100 IOs, 35854336 bytes 2024-09-17T23:20:32.490 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:200 IOs, 80808960 bytes 2024-09-17T23:20:32.603 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:300 IOs, 113293312 bytes 2024-09-17T23:20:32.703 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:400 IOs, 138062848 bytes 2024-09-17T23:20:32.852 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:500 IOs, 205217792 bytes 2024-09-17T23:20:32.964 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:600 IOs, 243599360 bytes 2024-09-17T23:20:33.074 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:700 IOs, 274466816 bytes 2024-09-17T23:20:33.157 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:800 IOs, 278470656 bytes 2024-09-17T23:20:33.288 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:900 IOs, 335765504 bytes 2024-09-17T23:20:33.388 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1000 IOs, 362781696 bytes 2024-09-17T23:20:33.513 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1100 IOs, 408571904 bytes 2024-09-17T23:20:33.667 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1200 IOs, 475709440 bytes 2024-09-17T23:20:33.760 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1300 IOs, 489267200 bytes 2024-09-17T23:20:33.903 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1400 IOs, 551856128 bytes 2024-09-17T23:20:34.010 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1500 IOs, 580911104 bytes 2024-09-17T23:20:34.092 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1600 IOs, 589985792 bytes 2024-09-17T23:20:34.185 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1700 IOs, 590609408 bytes 2024-09-17T23:20:34.314 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1800 IOs, 642281472 bytes 2024-09-17T23:20:34.428 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1900 IOs, 677275648 bytes 2024-09-17T23:20:34.510 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2000 IOs, 691908608 bytes 2024-09-17T23:20:34.620 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2100 IOs, 705847296 bytes 2024-09-17T23:20:34.928 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2200 IOs, 753785856 bytes 2024-09-17T23:20:35.050 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2300 IOs, 797051904 bytes 2024-09-17T23:20:35.190 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2400 IOs, 871895040 bytes 2024-09-17T23:20:35.313 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2500 IOs, 904404992 bytes 2024-09-17T23:20:35.638 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2600 IOs, 946994176 bytes 2024-09-17T23:20:35.770 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2700 IOs, 1003550720 bytes 2024-09-17T23:20:35.862 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2800 IOs, 1025328128 bytes 2024-09-17T23:20:35.973 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2900 IOs, 1048946688 bytes 2024-09-17T23:20:36.084 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3000 IOs, 1077281792 bytes 2024-09-17T23:20:36.217 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3100 IOs, 1134676992 bytes 2024-09-17T23:20:36.306 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3200 IOs, 1141797888 bytes 2024-09-17T23:20:36.400 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3300 IOs, 1156857856 bytes 2024-09-17T23:20:36.494 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3400 IOs, 1169811456 bytes 2024-09-17T23:20:36.620 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3500 IOs, 1212220416 bytes 2024-09-17T23:20:36.737 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3600 IOs, 1248280576 bytes 2024-09-17T23:20:36.867 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3700 IOs, 1298635776 bytes 2024-09-17T23:20:36.999 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3800 IOs, 1350867968 bytes 2024-09-17T23:20:37.083 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3900 IOs, 1351476224 bytes 2024-09-17T23:20:37.268 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4000 IOs, 1460922368 bytes 2024-09-17T23:20:37.369 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4100 IOs, 1484497920 bytes 2024-09-17T23:20:37.521 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4200 IOs, 1556606976 bytes 2024-09-17T23:20:37.652 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4300 IOs, 1603890176 bytes 2024-09-17T23:20:37.765 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4400 IOs, 1636818944 bytes 2024-09-17T23:20:37.860 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4500 IOs, 1654201344 bytes 2024-09-17T23:20:38.012 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4600 IOs, 1736659968 bytes 2024-09-17T23:20:38.135 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4700 IOs, 1767995392 bytes 2024-09-17T23:20:38.242 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4800 IOs, 1797316608 bytes 2024-09-17T23:20:38.358 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4900 IOs, 1838640128 bytes 2024-09-17T23:20:38.468 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5000 IOs, 1868961792 bytes 2024-09-17T23:20:38.609 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5100 IOs, 1929987072 bytes 2024-09-17T23:20:38.731 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5200 IOs, 1977312256 bytes 2024-09-17T23:20:38.891 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5300 IOs, 2046115840 bytes 2024-09-17T23:20:39.021 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5400 IOs, 2099260416 bytes 2024-09-17T23:20:39.133 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5500 IOs, 2134563840 bytes 2024-09-17T23:20:39.168 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ error_code=124 2024-09-17T23:20:39.169 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set -e 2024-09-17T23:20:39.169 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' 124 -eq 124 ']' 2024-09-17T23:20:39.169 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:20:39.169 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:20:39.169 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_26 2024-09-17T23:20:39.169 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_image_and_enable_mirror cluster2 mirror image_26 snapshot 128M 2024-09-17T23:20:39.169 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:20:39.169 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:20:39.169 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:20:39.169 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:20:39.169 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_26 2024-09-17T23:20:39.169 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:20:39.169 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=snapshot 2024-09-17T23:20:39.170 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n snapshot ']' 2024-09-17T23:20:39.170 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:20:39.170 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_image cluster2 mirror image_26 128M 2024-09-17T23:20:39.170 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:20:39.170 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:20:39.170 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:20:39.170 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:20:39.170 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_26 2024-09-17T23:20:39.170 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:20:39.170 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local size=128 2024-09-17T23:20:39.170 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n 128M ']' 2024-09-17T23:20:39.170 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ size=128M 2024-09-17T23:20:39.170 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:20:39.170 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_26 2024-09-17T23:20:39.219 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ is_pool_mirror_mode_image mirror 2024-09-17T23:20:39.219 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:20:39.219 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' image = image ']' 2024-09-17T23:20:39.219 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:20:39.219 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ enable_mirror cluster2 mirror image_26 snapshot 2024-09-17T23:20:39.219 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:20:39.219 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:20:39.220 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_26 2024-09-17T23:20:39.220 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=snapshot 2024-09-17T23:20:39.220 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_26 snapshot 2024-09-17T23:20:40.121 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Mirroring enabled 2024-09-17T23:20:40.133 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 info mirror/image_26 2024-09-17T23:20:40.174 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'image_26': 2024-09-17T23:20:40.174 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 128 MiB in 32 objects 2024-09-17T23:20:40.174 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:20:40.174 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 1 2024-09-17T23:20:40.174 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 13c1777413d5 2024-09-17T23:20:40.174 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.13c1777413d5 2024-09-17T23:20:40.174 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:20:40.174 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-17T23:20:40.174 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:20:40.174 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:20:40.174 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:20:39 2024 2024-09-17T23:20:40.175 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:20:39 2024 2024-09-17T23:20:40.175 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:20:39 2024 2024-09-17T23:20:40.175 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:20:40.175 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:20:40.175 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: abfb6ed3-8b33-429f-bf1d-a7fe92644887 2024-09-17T23:20:40.175 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: true 2024-09-17T23:20:40.177 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n '' ']' 2024-09-17T23:20:40.178 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ stress_write_image cluster2 mirror image_26 2024-09-17T23:20:40.178 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:20:40.178 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:20:40.178 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_26 2024-09-17T23:20:40.178 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-17T23:20:40.180 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local duration=8 2024-09-17T23:20:40.180 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set +e 2024-09-17T23:20:40.180 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_26 --debug-rbd=20 --debug-journaler=20 2024-09-17T23:20:40.375 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:100 IOs, 30793728 bytes 2024-09-17T23:20:40.467 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:200 IOs, 44110848 bytes 2024-09-17T23:20:40.585 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:300 IOs, 86412288 bytes 2024-09-17T23:20:40.681 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:400 IOs, 89965568 bytes 2024-09-17T23:20:40.796 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:500 IOs, 121784320 bytes 2024-09-17T23:20:40.947 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:600 IOs, 166295552 bytes 2024-09-17T23:20:41.063 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:700 IOs, 215842816 bytes 2024-09-17T23:20:41.164 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:800 IOs, 229001216 bytes 2024-09-17T23:20:41.324 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:900 IOs, 304091136 bytes 2024-09-17T23:20:41.432 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1000 IOs, 339139584 bytes 2024-09-17T23:20:41.628 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1100 IOs, 457117696 bytes 2024-09-17T23:20:41.742 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1200 IOs, 492312576 bytes 2024-09-17T23:20:41.864 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1300 IOs, 526101504 bytes 2024-09-17T23:20:42.029 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1400 IOs, 604346368 bytes 2024-09-17T23:20:42.140 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1500 IOs, 632291328 bytes 2024-09-17T23:20:42.282 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1600 IOs, 693253120 bytes 2024-09-17T23:20:42.415 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1700 IOs, 742075392 bytes 2024-09-17T23:20:42.513 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1800 IOs, 760247296 bytes 2024-09-17T23:20:42.640 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1900 IOs, 808423424 bytes 2024-09-17T23:20:42.745 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2000 IOs, 836892672 bytes 2024-09-17T23:20:42.892 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2100 IOs, 897559552 bytes 2024-09-17T23:20:42.996 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2200 IOs, 920338432 bytes 2024-09-17T23:20:43.075 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2300 IOs, 920947712 bytes 2024-09-17T23:20:43.208 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2400 IOs, 982974464 bytes 2024-09-17T23:20:43.340 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2500 IOs, 1029339136 bytes 2024-09-17T23:20:43.423 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2600 IOs, 1029976064 bytes 2024-09-17T23:20:43.556 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2700 IOs, 1082654720 bytes 2024-09-17T23:20:43.658 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2800 IOs, 1101961216 bytes 2024-09-17T23:20:43.824 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2900 IOs, 1182246912 bytes 2024-09-17T23:20:43.949 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3000 IOs, 1227584512 bytes 2024-09-17T23:20:44.037 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3100 IOs, 1247779840 bytes 2024-09-17T23:20:44.128 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3200 IOs, 1248403456 bytes 2024-09-17T23:20:44.218 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3300 IOs, 1257437184 bytes 2024-09-17T23:20:44.367 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3400 IOs, 1325922304 bytes 2024-09-17T23:20:44.522 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3500 IOs, 1397524480 bytes 2024-09-17T23:20:44.647 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3600 IOs, 1446309888 bytes 2024-09-17T23:20:44.741 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3700 IOs, 1459313664 bytes 2024-09-17T23:20:44.854 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3800 IOs, 1510343680 bytes 2024-09-17T23:20:44.988 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3900 IOs, 1565772800 bytes 2024-09-17T23:20:45.123 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4000 IOs, 1600266240 bytes 2024-09-17T23:20:45.202 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4100 IOs, 1600898048 bytes 2024-09-17T23:20:45.328 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4200 IOs, 1640928256 bytes 2024-09-17T23:20:45.437 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4300 IOs, 1666370560 bytes 2024-09-17T23:20:45.547 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4400 IOs, 1694398464 bytes 2024-09-17T23:20:45.672 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4500 IOs, 1741491200 bytes 2024-09-17T23:20:45.775 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4600 IOs, 1759449088 bytes 2024-09-17T23:20:45.924 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4700 IOs, 1807970304 bytes 2024-09-17T23:20:46.050 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4800 IOs, 1849367552 bytes 2024-09-17T23:20:46.165 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4900 IOs, 1886063616 bytes 2024-09-17T23:20:46.355 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5000 IOs, 1994762240 bytes 2024-09-17T23:20:46.488 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5100 IOs, 2039018496 bytes 2024-09-17T23:20:46.594 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5200 IOs, 2066347008 bytes 2024-09-17T23:20:46.712 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5300 IOs, 2105349120 bytes 2024-09-17T23:20:46.852 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5400 IOs, 2160034816 bytes 2024-09-17T23:20:46.953 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5500 IOs, 2182140928 bytes 2024-09-17T23:20:47.080 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5600 IOs, 2222307328 bytes 2024-09-17T23:20:47.189 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5700 IOs, 2253766656 bytes 2024-09-17T23:20:47.303 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5800 IOs, 2300902400 bytes 2024-09-17T23:20:47.437 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5900 IOs, 2340865024 bytes 2024-09-17T23:20:47.583 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6000 IOs, 2410448896 bytes 2024-09-17T23:20:47.703 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6100 IOs, 2449219584 bytes 2024-09-17T23:20:47.813 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6200 IOs, 2477576192 bytes 2024-09-17T23:20:47.930 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6300 IOs, 2526567424 bytes 2024-09-17T23:20:48.106 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6400 IOs, 2612847616 bytes 2024-09-17T23:20:48.185 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ error_code=124 2024-09-17T23:20:48.185 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set -e 2024-09-17T23:20:48.185 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' 124 -eq 124 ']' 2024-09-17T23:20:48.185 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:20:48.186 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:20:48.186 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_27 2024-09-17T23:20:48.186 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_image_and_enable_mirror cluster2 mirror image_27 snapshot 128M 2024-09-17T23:20:48.186 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:20:48.186 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:20:48.186 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:20:48.186 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:20:48.186 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_27 2024-09-17T23:20:48.186 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:20:48.186 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=snapshot 2024-09-17T23:20:48.186 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n snapshot ']' 2024-09-17T23:20:48.186 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:20:48.186 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_image cluster2 mirror image_27 128M 2024-09-17T23:20:48.187 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:20:48.187 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:20:48.187 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:20:48.188 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:20:48.188 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_27 2024-09-17T23:20:48.188 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:20:48.188 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local size=128 2024-09-17T23:20:48.188 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n 128M ']' 2024-09-17T23:20:48.188 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ size=128M 2024-09-17T23:20:48.188 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:20:48.188 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_27 2024-09-17T23:20:48.236 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ is_pool_mirror_mode_image mirror 2024-09-17T23:20:48.236 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:20:48.236 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' image = image ']' 2024-09-17T23:20:48.236 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:20:48.237 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ enable_mirror cluster2 mirror image_27 snapshot 2024-09-17T23:20:48.237 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:20:48.237 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:20:48.237 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_27 2024-09-17T23:20:48.237 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=snapshot 2024-09-17T23:20:48.237 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_27 snapshot 2024-09-17T23:20:49.143 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Mirroring enabled 2024-09-17T23:20:49.155 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 info mirror/image_27 2024-09-17T23:20:49.195 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'image_27': 2024-09-17T23:20:49.196 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 128 MiB in 32 objects 2024-09-17T23:20:49.196 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:20:49.196 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 1 2024-09-17T23:20:49.196 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 13cdbf96dc7c 2024-09-17T23:20:49.196 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.13cdbf96dc7c 2024-09-17T23:20:49.196 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:20:49.196 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-17T23:20:49.196 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:20:49.196 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:20:49.196 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:20:48 2024 2024-09-17T23:20:49.196 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:20:48 2024 2024-09-17T23:20:49.196 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:20:48 2024 2024-09-17T23:20:49.196 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:20:49.197 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:20:49.197 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: 3340f84d-5843-4769-b7b7-cd6b8eee8192 2024-09-17T23:20:49.197 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: true 2024-09-17T23:20:49.199 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n '' ']' 2024-09-17T23:20:49.199 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ stress_write_image cluster2 mirror image_27 2024-09-17T23:20:49.199 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:20:49.199 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:20:49.199 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_27 2024-09-17T23:20:49.199 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-17T23:20:49.201 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local duration=10 2024-09-17T23:20:49.201 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set +e 2024-09-17T23:20:49.201 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_27 --debug-rbd=20 --debug-journaler=20 2024-09-17T23:20:49.448 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:100 IOs, 81096704 bytes 2024-09-17T23:20:49.550 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:200 IOs, 98395136 bytes 2024-09-17T23:20:49.659 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:300 IOs, 127635456 bytes 2024-09-17T23:20:49.747 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:400 IOs, 140356608 bytes 2024-09-17T23:20:49.875 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:500 IOs, 194087936 bytes 2024-09-17T23:20:49.982 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:600 IOs, 215358464 bytes 2024-09-17T23:20:50.065 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:700 IOs, 218771456 bytes 2024-09-17T23:20:50.205 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:800 IOs, 280204288 bytes 2024-09-17T23:20:50.283 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:900 IOs, 280829952 bytes 2024-09-17T23:20:50.455 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1000 IOs, 388882432 bytes 2024-09-17T23:20:50.572 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1100 IOs, 421100544 bytes 2024-09-17T23:20:50.687 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1200 IOs, 458122240 bytes 2024-09-17T23:20:50.831 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1300 IOs, 524877824 bytes 2024-09-17T23:20:50.951 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1400 IOs, 563795968 bytes 2024-09-17T23:20:51.079 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1500 IOs, 631390208 bytes 2024-09-17T23:20:51.215 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1600 IOs, 659130368 bytes 2024-09-17T23:20:51.332 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1700 IOs, 706464768 bytes 2024-09-17T23:20:51.483 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1800 IOs, 772524032 bytes 2024-09-17T23:20:51.612 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1900 IOs, 832614400 bytes 2024-09-17T23:20:51.699 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2000 IOs, 838894592 bytes 2024-09-17T23:20:51.872 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2100 IOs, 926176256 bytes 2024-09-17T23:20:51.975 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2200 IOs, 944988160 bytes 2024-09-17T23:20:52.094 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2300 IOs, 985938944 bytes 2024-09-17T23:20:52.207 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2400 IOs, 1024541696 bytes 2024-09-17T23:20:52.321 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2500 IOs, 1057913856 bytes 2024-09-17T23:20:52.417 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2600 IOs, 1100134400 bytes 2024-09-17T23:20:52.615 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2700 IOs, 1200053248 bytes 2024-09-17T23:20:52.717 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2800 IOs, 1223047168 bytes 2024-09-17T23:20:52.883 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2900 IOs, 1314699264 bytes 2024-09-17T23:20:53.059 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3000 IOs, 1435535360 bytes 2024-09-17T23:20:53.170 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3100 IOs, 1437847552 bytes 2024-09-17T23:20:53.274 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3200 IOs, 1464265728 bytes 2024-09-17T23:20:53.364 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3300 IOs, 1473446912 bytes 2024-09-17T23:20:53.491 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3400 IOs, 1528053760 bytes 2024-09-17T23:20:53.595 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3500 IOs, 1552789504 bytes 2024-09-17T23:20:53.694 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3600 IOs, 1580860416 bytes 2024-09-17T23:20:53.831 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3700 IOs, 1632114688 bytes 2024-09-17T23:20:53.921 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3800 IOs, 1642449920 bytes 2024-09-17T23:20:54.034 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3900 IOs, 1694033920 bytes 2024-09-17T23:20:54.247 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4000 IOs, 1817083904 bytes 2024-09-17T23:20:54.421 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4100 IOs, 1903197184 bytes 2024-09-17T23:20:54.521 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4200 IOs, 1925390336 bytes 2024-09-17T23:20:54.680 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4300 IOs, 2008082432 bytes 2024-09-17T23:20:54.801 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4400 IOs, 2055742464 bytes 2024-09-17T23:20:54.916 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4500 IOs, 2093417472 bytes 2024-09-17T23:20:55.042 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4600 IOs, 2132467712 bytes 2024-09-17T23:20:55.169 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4700 IOs, 2191705088 bytes 2024-09-17T23:20:55.341 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4800 IOs, 2295806976 bytes 2024-09-17T23:20:55.496 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4900 IOs, 2351875072 bytes 2024-09-17T23:20:55.593 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5000 IOs, 2370131968 bytes 2024-09-17T23:20:55.701 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5100 IOs, 2404096000 bytes 2024-09-17T23:20:55.864 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5200 IOs, 2488310784 bytes 2024-09-17T23:20:55.994 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5300 IOs, 2531009536 bytes 2024-09-17T23:20:56.094 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5400 IOs, 2551772160 bytes 2024-09-17T23:20:56.231 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5500 IOs, 2607406080 bytes 2024-09-17T23:20:56.327 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5600 IOs, 2635149312 bytes 2024-09-17T23:20:56.475 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5700 IOs, 2696602624 bytes 2024-09-17T23:20:56.588 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5800 IOs, 2728699904 bytes 2024-09-17T23:20:56.695 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5900 IOs, 2759678976 bytes 2024-09-17T23:20:56.796 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6000 IOs, 2784915456 bytes 2024-09-17T23:20:56.909 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6100 IOs, 2813977600 bytes 2024-09-17T23:20:57.067 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6200 IOs, 2869539840 bytes 2024-09-17T23:20:57.175 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6300 IOs, 2912287744 bytes 2024-09-17T23:20:57.275 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6400 IOs, 2920410112 bytes 2024-09-17T23:20:57.586 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6500 IOs, 2954143744 bytes 2024-09-17T23:20:57.744 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6600 IOs, 3041054720 bytes 2024-09-17T23:20:57.844 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6700 IOs, 3060774912 bytes 2024-09-17T23:20:57.948 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6800 IOs, 3084378112 bytes 2024-09-17T23:20:58.062 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6900 IOs, 3126404096 bytes 2024-09-17T23:20:58.194 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7000 IOs, 3166115840 bytes 2024-09-17T23:20:58.364 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7100 IOs, 3263591424 bytes 2024-09-17T23:20:58.457 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7200 IOs, 3278642176 bytes 2024-09-17T23:20:58.567 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7300 IOs, 3307010048 bytes 2024-09-17T23:20:58.647 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7400 IOs, 3307606016 bytes 2024-09-17T23:20:58.747 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7500 IOs, 3344338944 bytes 2024-09-17T23:20:58.877 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7600 IOs, 3385402368 bytes 2024-09-17T23:20:58.961 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7700 IOs, 3386014720 bytes 2024-09-17T23:20:59.083 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7800 IOs, 3422309376 bytes 2024-09-17T23:20:59.208 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ error_code=124 2024-09-17T23:20:59.208 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set -e 2024-09-17T23:20:59.208 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' 124 -eq 124 ']' 2024-09-17T23:20:59.208 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:20:59.208 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:20:59.208 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_28 2024-09-17T23:20:59.208 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_image_and_enable_mirror cluster2 mirror image_28 snapshot 128M 2024-09-17T23:20:59.208 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:20:59.209 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:20:59.209 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:20:59.209 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:20:59.209 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_28 2024-09-17T23:20:59.210 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:20:59.210 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=snapshot 2024-09-17T23:20:59.210 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n snapshot ']' 2024-09-17T23:20:59.211 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:20:59.211 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_image cluster2 mirror image_28 128M 2024-09-17T23:20:59.211 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:20:59.211 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:20:59.211 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:20:59.211 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:20:59.211 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_28 2024-09-17T23:20:59.211 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:20:59.211 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local size=128 2024-09-17T23:20:59.211 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n 128M ']' 2024-09-17T23:20:59.211 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ size=128M 2024-09-17T23:20:59.211 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:20:59.211 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_28 2024-09-17T23:20:59.260 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ is_pool_mirror_mode_image mirror 2024-09-17T23:20:59.261 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:20:59.261 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' image = image ']' 2024-09-17T23:20:59.261 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:20:59.261 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ enable_mirror cluster2 mirror image_28 snapshot 2024-09-17T23:20:59.261 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:20:59.261 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:20:59.261 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_28 2024-09-17T23:20:59.261 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=snapshot 2024-09-17T23:20:59.261 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_28 snapshot 2024-09-17T23:21:00.159 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Mirroring enabled 2024-09-17T23:21:00.171 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 info mirror/image_28 2024-09-17T23:21:00.212 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'image_28': 2024-09-17T23:21:00.212 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 128 MiB in 32 objects 2024-09-17T23:21:00.212 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:21:00.213 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 1 2024-09-17T23:21:00.213 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 13d9a3ff5576 2024-09-17T23:21:00.213 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.13d9a3ff5576 2024-09-17T23:21:00.213 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:21:00.213 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-17T23:21:00.213 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:21:00.213 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:21:00.213 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:20:59 2024 2024-09-17T23:21:00.213 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:20:59 2024 2024-09-17T23:21:00.214 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:20:59 2024 2024-09-17T23:21:00.214 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:21:00.214 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:21:00.214 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: aa5b5df7-2c98-4b35-a04c-18d3b14fb4a4 2024-09-17T23:21:00.214 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: true 2024-09-17T23:21:00.216 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n '' ']' 2024-09-17T23:21:00.216 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ stress_write_image cluster2 mirror image_28 2024-09-17T23:21:00.216 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:21:00.216 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:21:00.216 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_28 2024-09-17T23:21:00.216 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-17T23:21:00.218 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local duration=10 2024-09-17T23:21:00.218 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set +e 2024-09-17T23:21:00.218 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_28 --debug-rbd=20 --debug-journaler=20 2024-09-17T23:21:00.410 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:100 IOs, 23519232 bytes 2024-09-17T23:21:00.557 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:200 IOs, 86155264 bytes 2024-09-17T23:21:00.684 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:300 IOs, 129506304 bytes 2024-09-17T23:21:00.822 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:400 IOs, 191310848 bytes 2024-09-17T23:21:00.960 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:500 IOs, 248321024 bytes 2024-09-17T23:21:01.139 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:600 IOs, 344225792 bytes 2024-09-17T23:21:01.225 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:700 IOs, 349744128 bytes 2024-09-17T23:21:01.321 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:800 IOs, 366172160 bytes 2024-09-17T23:21:01.433 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:900 IOs, 408535040 bytes 2024-09-17T23:21:01.527 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1000 IOs, 414994432 bytes 2024-09-17T23:21:01.652 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1100 IOs, 459734016 bytes 2024-09-17T23:21:01.770 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1200 IOs, 497977344 bytes 2024-09-17T23:21:01.877 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1300 IOs, 523609088 bytes 2024-09-17T23:21:01.988 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1400 IOs, 550307840 bytes 2024-09-17T23:21:02.102 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1500 IOs, 573795328 bytes 2024-09-17T23:21:02.271 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1600 IOs, 653484032 bytes 2024-09-17T23:21:02.334 INFO:tasks.ceph.cluster2.osd.2.smithi038.stderr:2024-09-17T23:21:02.331+0000 7f1c4b04a640 -1 reset not still connected to 0x561cc32db450 2024-09-17T23:21:02.334 INFO:tasks.ceph.cluster2.osd.2.smithi038.stderr:2024-09-17T23:21:02.331+0000 7f1c4b04a640 -1 reset not still connected to 0x561cc3a33380 2024-09-17T23:21:02.335 INFO:tasks.ceph.cluster2.osd.2.smithi038.stderr:2024-09-17T23:21:02.333+0000 7f1c4b04a640 -1 reset not still connected to 0x561cc3ec40d0 2024-09-17T23:21:02.336 INFO:tasks.ceph.cluster2.osd.2.smithi038.stderr:2024-09-17T23:21:02.333+0000 7f1c4b04a640 -1 reset not still connected to 0x561cc709a680 2024-09-17T23:21:02.337 INFO:tasks.ceph.cluster2.osd.2.smithi038.stderr:2024-09-17T23:21:02.335+0000 7f1c4b04a640 -1 reset not still connected to 0x561cccbcc8f0 2024-09-17T23:21:02.375 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1700 IOs, 669629440 bytes 2024-09-17T23:21:02.464 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1800 IOs, 678344704 bytes 2024-09-17T23:21:02.588 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1900 IOs, 721767424 bytes 2024-09-17T23:21:02.718 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2000 IOs, 770558976 bytes 2024-09-17T23:21:02.818 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2100 IOs, 793892864 bytes 2024-09-17T23:21:02.931 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2200 IOs, 825879552 bytes 2024-09-17T23:21:03.043 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2300 IOs, 854634496 bytes 2024-09-17T23:21:03.212 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2400 IOs, 937936896 bytes 2024-09-17T23:21:03.326 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2500 IOs, 995045376 bytes 2024-09-17T23:21:03.450 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2600 IOs, 1014707200 bytes 2024-09-17T23:21:03.576 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2700 IOs, 1067542528 bytes 2024-09-17T23:21:03.696 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2800 IOs, 1099070464 bytes 2024-09-17T23:21:03.823 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.5100111246821663) > weight (0.25) 2024-09-17T23:21:03.823 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.9836600437642773) > weight (0.25) 2024-09-17T23:21:03.824 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.6431706074362945) > weight (0.25) 2024-09-17T23:21:03.824 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.3 2024-09-17T23:21:03.824 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Sent signal 15 2024-09-17T23:21:03.824 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2024-09-17T23:21:03.826 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi038.stderr:2024-09-17T23:21:03.824+0000 7f638f1de640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 36673) UID: 1000 2024-09-17T23:21:03.861 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2900 IOs, 1188854784 bytes 2024-09-17T23:21:04.031 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3000 IOs, 1245971456 bytes 2024-09-17T23:21:04.144 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3100 IOs, 1260995584 bytes 2024-09-17T23:21:04.224 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3200 IOs, 1261583360 bytes 2024-09-17T23:21:04.318 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3300 IOs, 1269049344 bytes 2024-09-17T23:21:04.407 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3400 IOs, 1278126080 bytes 2024-09-17T23:21:04.553 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3500 IOs, 1344228352 bytes 2024-09-17T23:21:04.687 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3600 IOs, 1403816960 bytes 2024-09-17T23:21:04.777 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3700 IOs, 1405851648 bytes 2024-09-17T23:21:04.947 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3800 IOs, 1521110016 bytes 2024-09-17T23:21:05.072 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3900 IOs, 1541668864 bytes 2024-09-17T23:21:05.196 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4000 IOs, 1582131200 bytes 2024-09-17T23:21:05.304 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4100 IOs, 1614106624 bytes 2024-09-17T23:21:05.414 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4200 IOs, 1647132672 bytes 2024-09-17T23:21:05.602 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4300 IOs, 1743766528 bytes 2024-09-17T23:21:05.730 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4400 IOs, 1794354176 bytes 2024-09-17T23:21:05.859 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4500 IOs, 1853657088 bytes 2024-09-17T23:21:05.969 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4600 IOs, 1875849216 bytes 2024-09-17T23:21:06.068 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4700 IOs, 1886455808 bytes 2024-09-17T23:21:06.154 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4800 IOs, 1906763776 bytes 2024-09-17T23:21:06.258 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4900 IOs, 1910002688 bytes 2024-09-17T23:21:06.367 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5000 IOs, 1939894272 bytes 2024-09-17T23:21:06.557 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5100 IOs, 2051768320 bytes 2024-09-17T23:21:06.690 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5200 IOs, 2108259328 bytes 2024-09-17T23:21:06.844 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5300 IOs, 2184213504 bytes 2024-09-17T23:21:06.939 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5400 IOs, 2206560256 bytes 2024-09-17T23:21:07.035 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5500 IOs, 2211752960 bytes 2024-09-17T23:21:07.138 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5600 IOs, 2230851584 bytes 2024-09-17T23:21:07.260 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5700 IOs, 2281319424 bytes 2024-09-17T23:21:07.392 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5800 IOs, 2315834368 bytes 2024-09-17T23:21:07.488 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5900 IOs, 2334456832 bytes 2024-09-17T23:21:07.648 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6000 IOs, 2414569472 bytes 2024-09-17T23:21:07.824 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.3 2024-09-17T23:21:07.825 INFO:teuthology.orchestra.run:waiting for 600 2024-09-17T23:21:07.825 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.3 2024-09-17T23:21:07.825 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2024-09-17T23:21:07.825 DEBUG:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2024-09-17T23:21:07.828 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2024-09-17T23:21:07.828 INFO:tasks.rbd_mirror_thrash:waiting for 96 secs before thrashing 2024-09-17T23:21:08.001 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6100 IOs, 2487622656 bytes 2024-09-17T23:21:08.196 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6200 IOs, 2550160384 bytes 2024-09-17T23:21:08.379 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6300 IOs, 2623725568 bytes 2024-09-17T23:21:08.553 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6400 IOs, 2690200576 bytes 2024-09-17T23:21:08.643 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6500 IOs, 2710790144 bytes 2024-09-17T23:21:08.777 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6600 IOs, 2764565504 bytes 2024-09-17T23:21:08.909 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6700 IOs, 2812562432 bytes 2024-09-17T23:21:09.027 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6800 IOs, 2853633024 bytes 2024-09-17T23:21:09.155 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6900 IOs, 2896418816 bytes 2024-09-17T23:21:09.265 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7000 IOs, 2925805568 bytes 2024-09-17T23:21:09.426 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7100 IOs, 3006710784 bytes 2024-09-17T23:21:09.564 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7200 IOs, 3065906176 bytes 2024-09-17T23:21:09.679 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7300 IOs, 3104544768 bytes 2024-09-17T23:21:09.793 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7400 IOs, 3130533888 bytes 2024-09-17T23:21:09.900 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7500 IOs, 3161114624 bytes 2024-09-17T23:21:10.036 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7600 IOs, 3218693120 bytes 2024-09-17T23:21:10.152 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7700 IOs, 3250698240 bytes 2024-09-17T23:21:10.225 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ error_code=124 2024-09-17T23:21:10.225 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set -e 2024-09-17T23:21:10.225 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' 124 -eq 124 ']' 2024-09-17T23:21:10.225 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:21:10.225 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:21:10.225 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_29 2024-09-17T23:21:10.226 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_image_and_enable_mirror cluster2 mirror image_29 snapshot 128M 2024-09-17T23:21:10.226 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:21:10.226 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:21:10.226 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:21:10.226 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:21:10.226 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_29 2024-09-17T23:21:10.226 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:21:10.226 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=snapshot 2024-09-17T23:21:10.226 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n snapshot ']' 2024-09-17T23:21:10.226 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:21:10.226 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_image cluster2 mirror image_29 128M 2024-09-17T23:21:10.226 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:21:10.226 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:21:10.226 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:21:10.227 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:21:10.227 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_29 2024-09-17T23:21:10.227 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:21:10.227 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local size=128 2024-09-17T23:21:10.227 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n 128M ']' 2024-09-17T23:21:10.227 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ size=128M 2024-09-17T23:21:10.227 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:21:10.227 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_29 2024-09-17T23:21:10.277 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ is_pool_mirror_mode_image mirror 2024-09-17T23:21:10.277 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:21:10.277 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' image = image ']' 2024-09-17T23:21:10.277 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:21:10.278 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ enable_mirror cluster2 mirror image_29 snapshot 2024-09-17T23:21:10.278 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:21:10.278 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:21:10.278 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_29 2024-09-17T23:21:10.278 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=snapshot 2024-09-17T23:21:10.278 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_29 snapshot 2024-09-17T23:21:11.188 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Mirroring enabled 2024-09-17T23:21:11.201 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 info mirror/image_29 2024-09-17T23:21:11.241 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'image_29': 2024-09-17T23:21:11.241 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 128 MiB in 32 objects 2024-09-17T23:21:11.241 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:21:11.241 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 1 2024-09-17T23:21:11.241 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 13ebb8cd37ff 2024-09-17T23:21:11.241 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.13ebb8cd37ff 2024-09-17T23:21:11.242 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:21:11.242 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-17T23:21:11.242 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:21:11.242 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:21:11.242 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:21:10 2024 2024-09-17T23:21:11.242 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:21:10 2024 2024-09-17T23:21:11.242 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:21:10 2024 2024-09-17T23:21:11.242 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:21:11.242 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:21:11.242 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: 5ce3fe34-b38d-4dad-9e4f-b5cdc9405a96 2024-09-17T23:21:11.242 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: true 2024-09-17T23:21:11.244 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n '' ']' 2024-09-17T23:21:11.244 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ stress_write_image cluster2 mirror image_29 2024-09-17T23:21:11.245 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:21:11.245 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:21:11.245 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_29 2024-09-17T23:21:11.245 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-17T23:21:11.246 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local duration=7 2024-09-17T23:21:11.247 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set +e 2024-09-17T23:21:11.247 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_29 --debug-rbd=20 --debug-journaler=20 2024-09-17T23:21:11.423 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:100 IOs, 9708544 bytes 2024-09-17T23:21:11.505 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:200 IOs, 15108096 bytes 2024-09-17T23:21:11.595 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:300 IOs, 31626240 bytes 2024-09-17T23:21:11.723 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:400 IOs, 80351232 bytes 2024-09-17T23:21:11.849 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:500 IOs, 136664064 bytes 2024-09-17T23:21:11.954 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:600 IOs, 159210496 bytes 2024-09-17T23:21:12.119 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:700 IOs, 253605888 bytes 2024-09-17T23:21:12.229 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:800 IOs, 277327872 bytes 2024-09-17T23:21:12.308 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:900 IOs, 277963776 bytes 2024-09-17T23:21:12.482 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1000 IOs, 378000384 bytes 2024-09-17T23:21:12.626 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1100 IOs, 443265024 bytes 2024-09-17T23:21:12.727 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1200 IOs, 462038016 bytes 2024-09-17T23:21:12.840 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1300 IOs, 494764032 bytes 2024-09-17T23:21:12.954 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1400 IOs, 532979712 bytes 2024-09-17T23:21:13.084 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1500 IOs, 585648128 bytes 2024-09-17T23:21:13.201 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1600 IOs, 628730880 bytes 2024-09-17T23:21:13.348 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1700 IOs, 697496576 bytes 2024-09-17T23:21:13.466 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1800 IOs, 714972160 bytes 2024-09-17T23:21:13.588 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1900 IOs, 763165696 bytes 2024-09-17T23:21:13.704 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2000 IOs, 801256448 bytes 2024-09-17T23:21:13.808 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2100 IOs, 830713856 bytes 2024-09-17T23:21:13.919 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2200 IOs, 846440448 bytes 2024-09-17T23:21:14.067 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2300 IOs, 915579904 bytes 2024-09-17T23:21:14.233 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2400 IOs, 994252800 bytes 2024-09-17T23:21:14.369 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2500 IOs, 1043567616 bytes 2024-09-17T23:21:14.506 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2600 IOs, 1103068160 bytes 2024-09-17T23:21:14.614 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2700 IOs, 1131116544 bytes 2024-09-17T23:21:14.720 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2800 IOs, 1156998144 bytes 2024-09-17T23:21:14.811 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2900 IOs, 1181375488 bytes 2024-09-17T23:21:14.919 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3000 IOs, 1197305856 bytes 2024-09-17T23:21:15.031 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3100 IOs, 1228729344 bytes 2024-09-17T23:21:15.129 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3200 IOs, 1247831040 bytes 2024-09-17T23:21:15.228 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3300 IOs, 1266212864 bytes 2024-09-17T23:21:15.353 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3400 IOs, 1312061440 bytes 2024-09-17T23:21:15.463 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3500 IOs, 1331610624 bytes 2024-09-17T23:21:15.570 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3600 IOs, 1358636032 bytes 2024-09-17T23:21:15.695 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3700 IOs, 1403728896 bytes 2024-09-17T23:21:15.799 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3800 IOs, 1426958336 bytes 2024-09-17T23:21:15.896 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3900 IOs, 1439889408 bytes 2024-09-17T23:21:16.048 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4000 IOs, 1494910976 bytes 2024-09-17T23:21:16.127 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4100 IOs, 1495525376 bytes 2024-09-17T23:21:16.287 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4200 IOs, 1589484544 bytes 2024-09-17T23:21:16.425 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4300 IOs, 1629919232 bytes 2024-09-17T23:21:16.544 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4400 IOs, 1666401280 bytes 2024-09-17T23:21:16.697 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4500 IOs, 1742390272 bytes 2024-09-17T23:21:16.799 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4600 IOs, 1761188864 bytes 2024-09-17T23:21:16.945 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4700 IOs, 1833836544 bytes 2024-09-17T23:21:17.066 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4800 IOs, 1870802944 bytes 2024-09-17T23:21:17.182 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4900 IOs, 1928798208 bytes 2024-09-17T23:21:17.327 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5000 IOs, 1969611776 bytes 2024-09-17T23:21:17.458 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5100 IOs, 2019969024 bytes 2024-09-17T23:21:17.569 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5200 IOs, 2044095488 bytes 2024-09-17T23:21:17.699 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5300 IOs, 2095190016 bytes 2024-09-17T23:21:17.785 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5400 IOs, 2099974144 bytes 2024-09-17T23:21:17.977 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5500 IOs, 2223161344 bytes 2024-09-17T23:21:18.095 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5600 IOs, 2261126144 bytes 2024-09-17T23:21:18.190 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5700 IOs, 2261733376 bytes 2024-09-17T23:21:18.252 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ error_code=124 2024-09-17T23:21:18.253 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set -e 2024-09-17T23:21:18.253 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' 124 -eq 124 ']' 2024-09-17T23:21:18.253 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:21:18.253 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:21:18.253 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_30 2024-09-17T23:21:18.253 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_image_and_enable_mirror cluster2 mirror image_30 snapshot 128M 2024-09-17T23:21:18.253 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:21:18.253 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:21:18.253 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:21:18.253 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:21:18.253 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_30 2024-09-17T23:21:18.253 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:21:18.253 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=snapshot 2024-09-17T23:21:18.253 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n snapshot ']' 2024-09-17T23:21:18.254 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:21:18.254 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_image cluster2 mirror image_30 128M 2024-09-17T23:21:18.254 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:21:18.254 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:21:18.254 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:21:18.254 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:21:18.254 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_30 2024-09-17T23:21:18.254 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:21:18.254 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local size=128 2024-09-17T23:21:18.254 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n 128M ']' 2024-09-17T23:21:18.254 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ size=128M 2024-09-17T23:21:18.254 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:21:18.254 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_30 2024-09-17T23:21:18.303 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ is_pool_mirror_mode_image mirror 2024-09-17T23:21:18.303 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:21:18.303 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' image = image ']' 2024-09-17T23:21:18.303 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:21:18.303 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ enable_mirror cluster2 mirror image_30 snapshot 2024-09-17T23:21:18.303 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:21:18.303 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:21:18.303 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_30 2024-09-17T23:21:18.303 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=snapshot 2024-09-17T23:21:18.303 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_30 snapshot 2024-09-17T23:21:19.204 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Mirroring enabled 2024-09-17T23:21:19.215 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 info mirror/image_30 2024-09-17T23:21:19.256 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'image_30': 2024-09-17T23:21:19.256 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 128 MiB in 32 objects 2024-09-17T23:21:19.256 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:21:19.256 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 1 2024-09-17T23:21:19.256 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 13f733a54bc6 2024-09-17T23:21:19.257 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.13f733a54bc6 2024-09-17T23:21:19.257 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:21:19.257 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-17T23:21:19.257 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:21:19.257 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:21:19.257 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:21:18 2024 2024-09-17T23:21:19.257 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:21:18 2024 2024-09-17T23:21:19.257 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:21:18 2024 2024-09-17T23:21:19.257 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:21:19.257 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:21:19.257 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: c1039ee2-f5c2-417c-8ea6-8094c3f49b33 2024-09-17T23:21:19.257 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: true 2024-09-17T23:21:19.260 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n '' ']' 2024-09-17T23:21:19.260 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ stress_write_image cluster2 mirror image_30 2024-09-17T23:21:19.260 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:21:19.260 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:21:19.260 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_30 2024-09-17T23:21:19.260 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-17T23:21:19.262 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local duration=12 2024-09-17T23:21:19.262 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set +e 2024-09-17T23:21:19.262 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_30 --debug-rbd=20 --debug-journaler=20 2024-09-17T23:21:19.449 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:100 IOs, 23054336 bytes 2024-09-17T23:21:19.548 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:200 IOs, 42643456 bytes 2024-09-17T23:21:19.647 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:300 IOs, 65444864 bytes 2024-09-17T23:21:19.773 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:400 IOs, 114531328 bytes 2024-09-17T23:21:19.889 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:500 IOs, 166457344 bytes 2024-09-17T23:21:20.010 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:600 IOs, 194577408 bytes 2024-09-17T23:21:20.120 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:700 IOs, 224671744 bytes 2024-09-17T23:21:20.228 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:800 IOs, 252025856 bytes 2024-09-17T23:21:20.372 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:900 IOs, 309829632 bytes 2024-09-17T23:21:20.516 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1000 IOs, 380286976 bytes 2024-09-17T23:21:20.635 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1100 IOs, 401402880 bytes 2024-09-17T23:21:20.716 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1200 IOs, 407931904 bytes 2024-09-17T23:21:20.824 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1300 IOs, 437400576 bytes 2024-09-17T23:21:20.932 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1400 IOs, 464025600 bytes 2024-09-17T23:21:21.054 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1500 IOs, 494903296 bytes 2024-09-17T23:21:21.188 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1600 IOs, 546715648 bytes 2024-09-17T23:21:21.279 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1700 IOs, 553652224 bytes 2024-09-17T23:21:21.366 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1800 IOs, 554273792 bytes 2024-09-17T23:21:21.468 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1900 IOs, 575583232 bytes 2024-09-17T23:21:21.568 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2000 IOs, 590977024 bytes 2024-09-17T23:21:21.709 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2100 IOs, 657723392 bytes 2024-09-17T23:21:21.808 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2200 IOs, 666790912 bytes 2024-09-17T23:21:21.927 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2300 IOs, 709360640 bytes 2024-09-17T23:21:22.016 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2400 IOs, 715815936 bytes 2024-09-17T23:21:22.143 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2500 IOs, 771681280 bytes 2024-09-17T23:21:22.299 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2600 IOs, 841339904 bytes 2024-09-17T23:21:22.459 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2700 IOs, 915381248 bytes 2024-09-17T23:21:22.611 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2800 IOs, 990859264 bytes 2024-09-17T23:21:22.756 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2900 IOs, 1071138816 bytes 2024-09-17T23:21:22.897 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3000 IOs, 1119389696 bytes 2024-09-17T23:21:23.054 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3100 IOs, 1195456512 bytes 2024-09-17T23:21:23.173 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3200 IOs, 1238987776 bytes 2024-09-17T23:21:23.301 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3300 IOs, 1292797952 bytes 2024-09-17T23:21:23.394 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3400 IOs, 1296680960 bytes 2024-09-17T23:21:23.477 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3500 IOs, 1301178368 bytes 2024-09-17T23:21:23.565 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3600 IOs, 1309250560 bytes 2024-09-17T23:21:23.646 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3700 IOs, 1309860864 bytes 2024-09-17T23:21:23.792 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3800 IOs, 1416402944 bytes 2024-09-17T23:21:23.919 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3900 IOs, 1418156032 bytes 2024-09-17T23:21:24.037 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4000 IOs, 1455470592 bytes 2024-09-17T23:21:24.137 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4100 IOs, 1475922944 bytes 2024-09-17T23:21:24.226 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4200 IOs, 1485106176 bytes 2024-09-17T23:21:24.336 INFO:tasks.ceph.cluster2.osd.0.smithi038.stderr:2024-09-17T23:21:24.334+0000 7fa26a72e640 -1 reset not still connected to 0x55b328bc9c70 2024-09-17T23:21:24.362 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4300 IOs, 1540500480 bytes 2024-09-17T23:21:24.495 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4400 IOs, 1583603712 bytes 2024-09-17T23:21:24.607 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4500 IOs, 1614345216 bytes 2024-09-17T23:21:24.730 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4600 IOs, 1658564608 bytes 2024-09-17T23:21:24.863 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4700 IOs, 1714884608 bytes 2024-09-17T23:21:24.997 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4800 IOs, 1769078784 bytes 2024-09-17T23:21:25.121 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4900 IOs, 1823878144 bytes 2024-09-17T23:21:25.234 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5000 IOs, 1847004160 bytes 2024-09-17T23:21:25.331 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5100 IOs, 1866608640 bytes 2024-09-17T23:21:25.456 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5200 IOs, 1905094656 bytes 2024-09-17T23:21:25.562 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5300 IOs, 1930299392 bytes 2024-09-17T23:21:25.655 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5400 IOs, 1944794112 bytes 2024-09-17T23:21:25.817 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5500 IOs, 2029669376 bytes 2024-09-17T23:21:25.969 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5600 IOs, 2102560768 bytes 2024-09-17T23:21:26.086 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5700 IOs, 2141025280 bytes 2024-09-17T23:21:26.202 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5800 IOs, 2175014912 bytes 2024-09-17T23:21:26.313 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5900 IOs, 2210415616 bytes 2024-09-17T23:21:26.409 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6000 IOs, 2217830400 bytes 2024-09-17T23:21:26.587 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6100 IOs, 2313923584 bytes 2024-09-17T23:21:26.754 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6200 IOs, 2414420992 bytes 2024-09-17T23:21:26.915 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6300 IOs, 2477964288 bytes 2024-09-17T23:21:27.051 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6400 IOs, 2522835968 bytes 2024-09-17T23:21:27.152 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6500 IOs, 2545699840 bytes 2024-09-17T23:21:27.318 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6600 IOs, 2629574656 bytes 2024-09-17T23:21:27.423 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6700 IOs, 2655118336 bytes 2024-09-17T23:21:27.513 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6800 IOs, 2672939008 bytes 2024-09-17T23:21:27.640 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6900 IOs, 2701291520 bytes 2024-09-17T23:21:27.769 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7000 IOs, 2749111296 bytes 2024-09-17T23:21:27.884 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7100 IOs, 2789482496 bytes 2024-09-17T23:21:28.006 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7200 IOs, 2827740160 bytes 2024-09-17T23:21:28.107 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7300 IOs, 2846726144 bytes 2024-09-17T23:21:28.212 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7400 IOs, 2872446976 bytes 2024-09-17T23:21:28.382 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7500 IOs, 2963083264 bytes 2024-09-17T23:21:28.478 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7600 IOs, 2974885888 bytes 2024-09-17T23:21:28.579 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7700 IOs, 2991618048 bytes 2024-09-17T23:21:28.701 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7800 IOs, 3035183104 bytes 2024-09-17T23:21:28.880 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7900 IOs, 3137929216 bytes 2024-09-17T23:21:29.002 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8000 IOs, 3184163840 bytes 2024-09-17T23:21:29.088 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8100 IOs, 3213360128 bytes 2024-09-17T23:21:29.236 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8200 IOs, 3253354496 bytes 2024-09-17T23:21:29.414 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8300 IOs, 3374489600 bytes 2024-09-17T23:21:29.572 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8400 IOs, 3432327168 bytes 2024-09-17T23:21:29.726 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8500 IOs, 3521235968 bytes 2024-09-17T23:21:29.861 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8600 IOs, 3565817856 bytes 2024-09-17T23:21:29.966 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8700 IOs, 3602347008 bytes 2024-09-17T23:21:30.117 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8800 IOs, 3667596288 bytes 2024-09-17T23:21:30.275 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8900 IOs, 3766757376 bytes 2024-09-17T23:21:30.388 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9000 IOs, 3775270912 bytes 2024-09-17T23:21:30.502 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9100 IOs, 3808149504 bytes 2024-09-17T23:21:30.617 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9200 IOs, 3836593152 bytes 2024-09-17T23:21:30.756 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9300 IOs, 3903077376 bytes 2024-09-17T23:21:30.918 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9400 IOs, 3983789056 bytes 2024-09-17T23:21:31.013 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9500 IOs, 3995972608 bytes 2024-09-17T23:21:31.105 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9600 IOs, 4006262784 bytes 2024-09-17T23:21:31.251 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9700 IOs, 4076247040 bytes 2024-09-17T23:21:31.270 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ error_code=124 2024-09-17T23:21:31.271 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set -e 2024-09-17T23:21:31.271 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' 124 -eq 124 ']' 2024-09-17T23:21:31.271 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:21:31.271 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:21:31.271 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_31 2024-09-17T23:21:31.271 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_image_and_enable_mirror cluster2 mirror image_31 snapshot 128M 2024-09-17T23:21:31.271 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:21:31.271 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:21:31.271 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:21:31.271 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:21:31.271 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_31 2024-09-17T23:21:31.271 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:21:31.271 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=snapshot 2024-09-17T23:21:31.272 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n snapshot ']' 2024-09-17T23:21:31.272 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:21:31.272 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_image cluster2 mirror image_31 128M 2024-09-17T23:21:31.272 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:21:31.272 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:21:31.272 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:21:31.272 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:21:31.272 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_31 2024-09-17T23:21:31.272 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:21:31.272 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local size=128 2024-09-17T23:21:31.272 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n 128M ']' 2024-09-17T23:21:31.272 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ size=128M 2024-09-17T23:21:31.272 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:21:31.273 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_31 2024-09-17T23:21:31.325 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ is_pool_mirror_mode_image mirror 2024-09-17T23:21:31.325 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:21:31.325 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' image = image ']' 2024-09-17T23:21:31.325 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:21:31.326 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ enable_mirror cluster2 mirror image_31 snapshot 2024-09-17T23:21:31.326 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:21:31.326 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:21:31.326 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_31 2024-09-17T23:21:31.326 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=snapshot 2024-09-17T23:21:31.326 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_31 snapshot 2024-09-17T23:21:31.863 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Mirroring enabled 2024-09-17T23:21:31.873 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 info mirror/image_31 2024-09-17T23:21:31.915 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'image_31': 2024-09-17T23:21:31.915 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 128 MiB in 32 objects 2024-09-17T23:21:31.915 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:21:31.915 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 1 2024-09-17T23:21:31.915 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 1403e8f9ae01 2024-09-17T23:21:31.915 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.1403e8f9ae01 2024-09-17T23:21:31.915 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:21:31.915 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-17T23:21:31.915 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:21:31.915 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:21:31.915 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:21:31 2024 2024-09-17T23:21:31.915 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:21:31 2024 2024-09-17T23:21:31.916 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:21:31 2024 2024-09-17T23:21:31.916 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:21:31.916 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:21:31.916 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: c8ba8555-2fc3-405e-b6a5-7ff5d6a4334c 2024-09-17T23:21:31.916 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: true 2024-09-17T23:21:31.918 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n '' ']' 2024-09-17T23:21:31.918 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ stress_write_image cluster2 mirror image_31 2024-09-17T23:21:31.918 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:21:31.919 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:21:31.919 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_31 2024-09-17T23:21:31.919 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-17T23:21:31.920 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local duration=11 2024-09-17T23:21:31.920 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set +e 2024-09-17T23:21:31.921 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_31 --debug-rbd=20 --debug-journaler=20 2024-09-17T23:21:32.119 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:100 IOs, 40943616 bytes 2024-09-17T23:21:32.240 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:200 IOs, 73476096 bytes 2024-09-17T23:21:32.326 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:300 IOs, 89818112 bytes 2024-09-17T23:21:32.464 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:400 IOs, 139611136 bytes 2024-09-17T23:21:32.586 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:500 IOs, 185507840 bytes 2024-09-17T23:21:32.725 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:600 IOs, 241798144 bytes 2024-09-17T23:21:32.849 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:700 IOs, 285454336 bytes 2024-09-17T23:21:32.955 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:800 IOs, 311951360 bytes 2024-09-17T23:21:33.049 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:900 IOs, 328694784 bytes 2024-09-17T23:21:33.196 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1000 IOs, 397374464 bytes 2024-09-17T23:21:33.330 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1100 IOs, 450603008 bytes 2024-09-17T23:21:33.425 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1200 IOs, 462904320 bytes 2024-09-17T23:21:33.552 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1300 IOs, 504823808 bytes 2024-09-17T23:21:33.656 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1400 IOs, 527432704 bytes 2024-09-17T23:21:33.791 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1500 IOs, 582476800 bytes 2024-09-17T23:21:33.896 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1600 IOs, 605437952 bytes 2024-09-17T23:21:34.009 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1700 IOs, 628101120 bytes 2024-09-17T23:21:34.090 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1800 IOs, 629270528 bytes 2024-09-17T23:21:34.251 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1900 IOs, 709350400 bytes 2024-09-17T23:21:34.348 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2000 IOs, 727527424 bytes 2024-09-17T23:21:34.456 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2100 IOs, 758894592 bytes 2024-09-17T23:21:34.586 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2200 IOs, 800789504 bytes 2024-09-17T23:21:34.681 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2300 IOs, 815095808 bytes 2024-09-17T23:21:34.772 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2400 IOs, 828104704 bytes 2024-09-17T23:21:34.893 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2500 IOs, 869692416 bytes 2024-09-17T23:21:34.991 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2600 IOs, 906043392 bytes 2024-09-17T23:21:35.136 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2700 IOs, 947227648 bytes 2024-09-17T23:21:35.235 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2800 IOs, 970388480 bytes 2024-09-17T23:21:35.391 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2900 IOs, 1056733184 bytes 2024-09-17T23:21:35.528 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3000 IOs, 1109593088 bytes 2024-09-17T23:21:35.667 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3100 IOs, 1173441536 bytes 2024-09-17T23:21:35.779 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3200 IOs, 1192564736 bytes 2024-09-17T23:21:35.899 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3300 IOs, 1236295680 bytes 2024-09-17T23:21:36.034 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3400 IOs, 1289184256 bytes 2024-09-17T23:21:36.151 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3500 IOs, 1328530432 bytes 2024-09-17T23:21:36.253 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3600 IOs, 1351758848 bytes 2024-09-17T23:21:36.367 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3700 IOs, 1384844288 bytes 2024-09-17T23:21:36.460 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3800 IOs, 1396512768 bytes 2024-09-17T23:21:36.635 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3900 IOs, 1493997568 bytes 2024-09-17T23:21:36.722 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4000 IOs, 1521007616 bytes 2024-09-17T23:21:36.873 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4100 IOs, 1566595072 bytes 2024-09-17T23:21:37.028 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4200 IOs, 1642977280 bytes 2024-09-17T23:21:37.141 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4300 IOs, 1675856896 bytes 2024-09-17T23:21:37.244 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4400 IOs, 1699272704 bytes 2024-09-17T23:21:37.397 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4500 IOs, 1770344448 bytes 2024-09-17T23:21:37.502 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4600 IOs, 1797262336 bytes 2024-09-17T23:21:37.604 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4700 IOs, 1818745856 bytes 2024-09-17T23:21:37.767 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4800 IOs, 1898501120 bytes 2024-09-17T23:21:37.866 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4900 IOs, 1935962112 bytes 2024-09-17T23:21:38.022 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5000 IOs, 2002159616 bytes 2024-09-17T23:21:38.156 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5100 IOs, 2062448640 bytes 2024-09-17T23:21:38.266 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5200 IOs, 2078184448 bytes 2024-09-17T23:21:38.370 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5300 IOs, 2100681728 bytes 2024-09-17T23:21:38.466 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5400 IOs, 2119265280 bytes 2024-09-17T23:21:38.602 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5500 IOs, 2170959872 bytes 2024-09-17T23:21:38.695 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5600 IOs, 2183585792 bytes 2024-09-17T23:21:38.826 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5700 IOs, 2232666112 bytes 2024-09-17T23:21:38.962 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5800 IOs, 2291274752 bytes 2024-09-17T23:21:39.115 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5900 IOs, 2370052096 bytes 2024-09-17T23:21:39.197 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6000 IOs, 2395063296 bytes 2024-09-17T23:21:39.357 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6100 IOs, 2455038976 bytes 2024-09-17T23:21:39.465 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6200 IOs, 2485839872 bytes 2024-09-17T23:21:39.584 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6300 IOs, 2519754752 bytes 2024-09-17T23:21:39.690 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6400 IOs, 2545486848 bytes 2024-09-17T23:21:39.830 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6500 IOs, 2609397760 bytes 2024-09-17T23:21:39.937 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6600 IOs, 2628744192 bytes 2024-09-17T23:21:40.036 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6700 IOs, 2655587328 bytes 2024-09-17T23:21:40.178 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6800 IOs, 2720249856 bytes 2024-09-17T23:21:40.288 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6900 IOs, 2751206400 bytes 2024-09-17T23:21:40.415 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7000 IOs, 2789020672 bytes 2024-09-17T23:21:40.496 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7100 IOs, 2789652480 bytes 2024-09-17T23:21:40.613 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7200 IOs, 2826962944 bytes 2024-09-17T23:21:40.756 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7300 IOs, 2885380096 bytes 2024-09-17T23:21:40.866 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7400 IOs, 2912523264 bytes 2024-09-17T23:21:40.974 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7500 IOs, 2942871552 bytes 2024-09-17T23:21:41.167 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7600 IOs, 3055399936 bytes 2024-09-17T23:21:41.253 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7700 IOs, 3066441728 bytes 2024-09-17T23:21:41.347 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7800 IOs, 3078740992 bytes 2024-09-17T23:21:41.446 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7900 IOs, 3093931008 bytes 2024-09-17T23:21:41.583 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8000 IOs, 3153592320 bytes 2024-09-17T23:21:41.700 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8100 IOs, 3180051456 bytes 2024-09-17T23:21:41.802 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8200 IOs, 3226049536 bytes 2024-09-17T23:21:41.916 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8300 IOs, 3233709056 bytes 2024-09-17T23:21:42.011 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8400 IOs, 3249418240 bytes 2024-09-17T23:21:42.122 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8500 IOs, 3279875072 bytes 2024-09-17T23:21:42.203 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8600 IOs, 3280540672 bytes 2024-09-17T23:21:42.297 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8700 IOs, 3290980352 bytes 2024-09-17T23:21:42.392 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8800 IOs, 3306650624 bytes 2024-09-17T23:21:42.512 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8900 IOs, 3344283648 bytes 2024-09-17T23:21:42.597 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9000 IOs, 3348882432 bytes 2024-09-17T23:21:42.697 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9100 IOs, 3364352000 bytes 2024-09-17T23:21:42.798 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9200 IOs, 3383145472 bytes 2024-09-17T23:21:42.928 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ error_code=124 2024-09-17T23:21:42.928 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set -e 2024-09-17T23:21:42.928 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' 124 -eq 124 ']' 2024-09-17T23:21:42.928 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:21:42.929 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:21:42.929 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_32 2024-09-17T23:21:42.929 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_image_and_enable_mirror cluster2 mirror image_32 snapshot 128M 2024-09-17T23:21:42.929 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:21:42.929 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:21:42.929 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:21:42.929 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:21:42.929 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_32 2024-09-17T23:21:42.929 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:21:42.929 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=snapshot 2024-09-17T23:21:42.929 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n snapshot ']' 2024-09-17T23:21:42.929 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:21:42.929 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_image cluster2 mirror image_32 128M 2024-09-17T23:21:42.929 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:21:42.930 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:21:42.930 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:21:42.930 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:21:42.930 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_32 2024-09-17T23:21:42.930 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:21:42.930 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local size=128 2024-09-17T23:21:42.930 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n 128M ']' 2024-09-17T23:21:42.930 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ size=128M 2024-09-17T23:21:42.930 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:21:42.930 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_32 2024-09-17T23:21:42.979 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ is_pool_mirror_mode_image mirror 2024-09-17T23:21:42.979 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:21:42.980 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' image = image ']' 2024-09-17T23:21:42.980 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:21:42.980 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ enable_mirror cluster2 mirror image_32 snapshot 2024-09-17T23:21:42.980 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:21:42.980 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:21:42.980 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_32 2024-09-17T23:21:42.980 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=snapshot 2024-09-17T23:21:42.980 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_32 snapshot 2024-09-17T23:21:43.883 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Mirroring enabled 2024-09-17T23:21:43.895 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 info mirror/image_32 2024-09-17T23:21:43.935 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'image_32': 2024-09-17T23:21:43.935 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 128 MiB in 32 objects 2024-09-17T23:21:43.935 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:21:43.935 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 1 2024-09-17T23:21:43.935 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 140f9be65796 2024-09-17T23:21:43.935 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.140f9be65796 2024-09-17T23:21:43.936 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:21:43.936 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-17T23:21:43.936 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:21:43.936 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:21:43.936 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:21:42 2024 2024-09-17T23:21:43.936 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:21:42 2024 2024-09-17T23:21:43.936 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:21:42 2024 2024-09-17T23:21:43.936 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:21:43.936 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:21:43.936 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: c47737c4-4126-4b7d-8eff-21c600ab88e3 2024-09-17T23:21:43.936 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: true 2024-09-17T23:21:43.939 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n '' ']' 2024-09-17T23:21:43.939 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ stress_write_image cluster2 mirror image_32 2024-09-17T23:21:43.939 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:21:43.939 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:21:43.939 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_32 2024-09-17T23:21:43.939 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-17T23:21:43.941 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local duration=9 2024-09-17T23:21:43.941 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set +e 2024-09-17T23:21:43.941 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_32 --debug-rbd=20 --debug-journaler=20 2024-09-17T23:21:44.129 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:100 IOs, 18844672 bytes 2024-09-17T23:21:44.234 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:200 IOs, 42708992 bytes 2024-09-17T23:21:44.356 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:300 IOs, 94903296 bytes 2024-09-17T23:21:44.464 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:400 IOs, 117628928 bytes 2024-09-17T23:21:44.547 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:500 IOs, 118249472 bytes 2024-09-17T23:21:44.660 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:600 IOs, 148649984 bytes 2024-09-17T23:21:44.781 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:700 IOs, 191629312 bytes 2024-09-17T23:21:44.890 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:800 IOs, 211499008 bytes 2024-09-17T23:21:44.988 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:900 IOs, 231052288 bytes 2024-09-17T23:21:45.100 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1000 IOs, 262989824 bytes 2024-09-17T23:21:45.236 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1100 IOs, 319948800 bytes 2024-09-17T23:21:45.331 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1200 IOs, 332635136 bytes 2024-09-17T23:21:45.456 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1300 IOs, 374619136 bytes 2024-09-17T23:21:45.568 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1400 IOs, 408643584 bytes 2024-09-17T23:21:45.683 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1500 IOs, 441289728 bytes 2024-09-17T23:21:45.800 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1600 IOs, 474555392 bytes 2024-09-17T23:21:45.912 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1700 IOs, 496294912 bytes 2024-09-17T23:21:46.087 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1800 IOs, 537695232 bytes 2024-09-17T23:21:46.271 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1900 IOs, 651031552 bytes 2024-09-17T23:21:46.386 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2000 IOs, 672898048 bytes 2024-09-17T23:21:46.501 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2100 IOs, 709682176 bytes 2024-09-17T23:21:46.625 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2200 IOs, 748597248 bytes 2024-09-17T23:21:46.727 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2300 IOs, 771321856 bytes 2024-09-17T23:21:46.833 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2400 IOs, 788325376 bytes 2024-09-17T23:21:46.976 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2500 IOs, 851126272 bytes 2024-09-17T23:21:47.123 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2600 IOs, 921778176 bytes 2024-09-17T23:21:47.221 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2700 IOs, 938254336 bytes 2024-09-17T23:21:47.344 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2800 IOs, 978837504 bytes 2024-09-17T23:21:47.425 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2900 IOs, 979471360 bytes 2024-09-17T23:21:47.524 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3000 IOs, 995322880 bytes 2024-09-17T23:21:47.660 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3100 IOs, 1050467328 bytes 2024-09-17T23:21:47.742 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3200 IOs, 1051083776 bytes 2024-09-17T23:21:47.902 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3300 IOs, 1125679104 bytes 2024-09-17T23:21:48.050 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3400 IOs, 1192656896 bytes 2024-09-17T23:21:48.159 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3500 IOs, 1221678080 bytes 2024-09-17T23:21:48.272 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3600 IOs, 1256850432 bytes 2024-09-17T23:21:48.359 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3700 IOs, 1268403200 bytes 2024-09-17T23:21:48.478 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3800 IOs, 1303081984 bytes 2024-09-17T23:21:48.591 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3900 IOs, 1335363584 bytes 2024-09-17T23:21:48.695 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4000 IOs, 1354784768 bytes 2024-09-17T23:21:48.809 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4100 IOs, 1392123904 bytes 2024-09-17T23:21:48.986 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4200 IOs, 1488593920 bytes 2024-09-17T23:21:49.092 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4300 IOs, 1503283200 bytes 2024-09-17T23:21:49.209 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4400 IOs, 1559186432 bytes 2024-09-17T23:21:49.348 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4500 IOs, 1597700096 bytes 2024-09-17T23:21:49.468 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4600 IOs, 1639075840 bytes 2024-09-17T23:21:49.576 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4700 IOs, 1667506176 bytes 2024-09-17T23:21:49.690 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4800 IOs, 1694470144 bytes 2024-09-17T23:21:49.781 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4900 IOs, 1707586560 bytes 2024-09-17T23:21:49.889 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5000 IOs, 1730595840 bytes 2024-09-17T23:21:50.060 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5100 IOs, 1817649152 bytes 2024-09-17T23:21:50.212 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5200 IOs, 1895037952 bytes 2024-09-17T23:21:50.347 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5300 IOs, 1948693504 bytes 2024-09-17T23:21:50.535 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5400 IOs, 2063613952 bytes 2024-09-17T23:21:50.689 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5500 IOs, 2139635712 bytes 2024-09-17T23:21:50.814 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5600 IOs, 2183123968 bytes 2024-09-17T23:21:51.017 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5700 IOs, 2238875648 bytes 2024-09-17T23:21:51.191 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5800 IOs, 2274911232 bytes 2024-09-17T23:21:51.344 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5900 IOs, 2338698240 bytes 2024-09-17T23:21:51.445 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6000 IOs, 2367020032 bytes 2024-09-17T23:21:51.559 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6100 IOs, 2407332864 bytes 2024-09-17T23:21:51.694 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6200 IOs, 2441158656 bytes 2024-09-17T23:21:51.845 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6300 IOs, 2534175744 bytes 2024-09-17T23:21:52.058 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6400 IOs, 2644368384 bytes 2024-09-17T23:21:52.230 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6500 IOs, 2740491264 bytes 2024-09-17T23:21:52.345 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6600 IOs, 2776894464 bytes 2024-09-17T23:21:52.474 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6700 IOs, 2836937728 bytes 2024-09-17T23:21:52.585 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6800 IOs, 2856307712 bytes 2024-09-17T23:21:52.737 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6900 IOs, 2942979072 bytes 2024-09-17T23:21:52.891 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7000 IOs, 2990909440 bytes 2024-09-17T23:21:52.949 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ error_code=124 2024-09-17T23:21:52.949 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set -e 2024-09-17T23:21:52.949 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' 124 -eq 124 ']' 2024-09-17T23:21:52.949 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:21:52.949 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:21:52.949 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_33 2024-09-17T23:21:52.949 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_image_and_enable_mirror cluster2 mirror image_33 snapshot 128M 2024-09-17T23:21:52.949 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:21:52.949 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:21:52.949 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:21:52.949 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:21:52.950 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_33 2024-09-17T23:21:52.950 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:21:52.950 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=snapshot 2024-09-17T23:21:52.950 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n snapshot ']' 2024-09-17T23:21:52.950 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:21:52.950 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_image cluster2 mirror image_33 128M 2024-09-17T23:21:52.950 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:21:52.950 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:21:52.950 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:21:52.950 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:21:52.950 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_33 2024-09-17T23:21:52.950 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:21:52.950 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local size=128 2024-09-17T23:21:52.950 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n 128M ']' 2024-09-17T23:21:52.950 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ size=128M 2024-09-17T23:21:52.950 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:21:52.951 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_33 2024-09-17T23:21:52.999 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ is_pool_mirror_mode_image mirror 2024-09-17T23:21:53.000 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:21:53.000 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' image = image ']' 2024-09-17T23:21:53.000 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:21:53.000 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ enable_mirror cluster2 mirror image_33 snapshot 2024-09-17T23:21:53.000 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:21:53.000 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:21:53.000 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_33 2024-09-17T23:21:53.000 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=snapshot 2024-09-17T23:21:53.000 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_33 snapshot 2024-09-17T23:21:53.910 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Mirroring enabled 2024-09-17T23:21:53.922 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 info mirror/image_33 2024-09-17T23:21:53.935 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.29552736136980096) > weight (0.25) 2024-09-17T23:21:53.935 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.9925687969556392) > weight (0.25) 2024-09-17T23:21:53.935 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.2662778221777753) > weight (0.25) 2024-09-17T23:21:53.935 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.25117652599745266) > weight (0.25) 2024-09-17T23:21:53.935 INFO:tasks.rbd_mirror_thrash:waiting for 116 secs before thrashing 2024-09-17T23:21:54.163 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'image_33': 2024-09-17T23:21:54.164 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 128 MiB in 32 objects 2024-09-17T23:21:54.164 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:21:54.164 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 1 2024-09-17T23:21:54.164 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 141b5491984 2024-09-17T23:21:54.164 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.141b5491984 2024-09-17T23:21:54.164 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:21:54.164 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-17T23:21:54.164 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:21:54.164 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:21:54.164 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:21:52 2024 2024-09-17T23:21:54.164 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:21:52 2024 2024-09-17T23:21:54.164 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:21:52 2024 2024-09-17T23:21:54.165 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:21:54.165 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:21:54.165 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: d3b01291-4a3c-45c8-9b7e-71a70aa71cd3 2024-09-17T23:21:54.165 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: true 2024-09-17T23:21:54.167 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n '' ']' 2024-09-17T23:21:54.168 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ stress_write_image cluster2 mirror image_33 2024-09-17T23:21:54.168 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:21:54.168 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:21:54.168 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_33 2024-09-17T23:21:54.168 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-17T23:21:54.170 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local duration=13 2024-09-17T23:21:54.170 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set +e 2024-09-17T23:21:54.170 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_33 --debug-rbd=20 --debug-journaler=20 2024-09-17T23:21:54.422 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:100 IOs, 84111360 bytes 2024-09-17T23:21:54.550 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:200 IOs, 129670144 bytes 2024-09-17T23:21:54.698 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:300 IOs, 201435136 bytes 2024-09-17T23:21:54.820 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:400 IOs, 239002624 bytes 2024-09-17T23:21:54.973 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:500 IOs, 313972736 bytes 2024-09-17T23:21:55.065 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:600 IOs, 317092864 bytes 2024-09-17T23:21:55.183 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:700 IOs, 350604288 bytes 2024-09-17T23:21:55.269 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:800 IOs, 352581632 bytes 2024-09-17T23:21:55.380 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:900 IOs, 398166016 bytes 2024-09-17T23:21:55.519 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1000 IOs, 441409536 bytes 2024-09-17T23:21:55.669 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1100 IOs, 511965184 bytes 2024-09-17T23:21:55.770 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1200 IOs, 530244608 bytes 2024-09-17T23:21:55.931 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1300 IOs, 616752128 bytes 2024-09-17T23:21:56.037 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1400 IOs, 639282176 bytes 2024-09-17T23:21:56.231 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1500 IOs, 749762560 bytes 2024-09-17T23:21:56.356 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1600 IOs, 800506880 bytes 2024-09-17T23:21:56.487 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1700 IOs, 836641792 bytes 2024-09-17T23:21:56.608 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1800 IOs, 877549568 bytes 2024-09-17T23:21:56.736 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1900 IOs, 941531136 bytes 2024-09-17T23:21:56.883 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2000 IOs, 994494464 bytes 2024-09-17T23:21:56.989 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2100 IOs, 1011777536 bytes 2024-09-17T23:21:57.096 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2200 IOs, 1022436352 bytes 2024-09-17T23:21:57.249 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2300 IOs, 1089158144 bytes 2024-09-17T23:21:57.352 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2400 IOs, 1110369280 bytes 2024-09-17T23:21:57.453 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2500 IOs, 1129895936 bytes 2024-09-17T23:21:57.553 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2600 IOs, 1150536704 bytes 2024-09-17T23:21:57.638 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2700 IOs, 1153552384 bytes 2024-09-17T23:21:57.764 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2800 IOs, 1202150400 bytes 2024-09-17T23:21:57.873 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2900 IOs, 1219485696 bytes 2024-09-17T23:21:57.992 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3000 IOs, 1248090112 bytes 2024-09-17T23:21:58.103 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3100 IOs, 1275056128 bytes 2024-09-17T23:21:58.239 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3200 IOs, 1329440768 bytes 2024-09-17T23:21:58.359 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3300 IOs, 1360145408 bytes 2024-09-17T23:21:58.456 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3400 IOs, 1375023104 bytes 2024-09-17T23:21:58.614 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3500 IOs, 1465734144 bytes 2024-09-17T23:21:58.730 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3600 IOs, 1494364160 bytes 2024-09-17T23:21:58.837 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3700 IOs, 1533916160 bytes 2024-09-17T23:21:58.959 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3800 IOs, 1555919872 bytes 2024-09-17T23:21:59.053 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3900 IOs, 1571074048 bytes 2024-09-17T23:21:59.191 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4000 IOs, 1628867584 bytes 2024-09-17T23:21:59.295 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4100 IOs, 1639054336 bytes 2024-09-17T23:21:59.377 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4200 IOs, 1640612864 bytes 2024-09-17T23:21:59.533 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4300 IOs, 1714484224 bytes 2024-09-17T23:21:59.656 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4400 IOs, 1759134720 bytes 2024-09-17T23:21:59.805 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4500 IOs, 1841703936 bytes 2024-09-17T23:21:59.943 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4600 IOs, 1882884096 bytes 2024-09-17T23:22:00.068 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4700 IOs, 1920749568 bytes 2024-09-17T23:22:00.197 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4800 IOs, 1966695424 bytes 2024-09-17T23:22:00.301 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4900 IOs, 1984745472 bytes 2024-09-17T23:22:00.405 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5000 IOs, 2006886400 bytes 2024-09-17T23:22:00.517 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5100 IOs, 2039865344 bytes 2024-09-17T23:22:00.624 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5200 IOs, 2066643968 bytes 2024-09-17T23:22:00.808 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5300 IOs, 2171585536 bytes 2024-09-17T23:22:00.947 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5400 IOs, 2224603136 bytes 2024-09-17T23:22:01.034 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5500 IOs, 2225215488 bytes 2024-09-17T23:22:01.198 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5600 IOs, 2308253696 bytes 2024-09-17T23:22:01.299 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5700 IOs, 2327184384 bytes 2024-09-17T23:22:01.406 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5800 IOs, 2345092096 bytes 2024-09-17T23:22:01.501 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5900 IOs, 2362105856 bytes 2024-09-17T23:22:01.627 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6000 IOs, 2407273472 bytes 2024-09-17T23:22:01.761 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6100 IOs, 2453111808 bytes 2024-09-17T23:22:01.953 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6200 IOs, 2563325952 bytes 2024-09-17T23:22:02.139 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6300 IOs, 2673772544 bytes 2024-09-17T23:22:02.256 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6400 IOs, 2700288000 bytes 2024-09-17T23:22:02.376 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6500 IOs, 2729751552 bytes 2024-09-17T23:22:02.484 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6600 IOs, 2755318784 bytes 2024-09-17T23:22:02.617 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6700 IOs, 2809368576 bytes 2024-09-17T23:22:02.748 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6800 IOs, 2858484736 bytes 2024-09-17T23:22:02.858 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6900 IOs, 2887281664 bytes 2024-09-17T23:22:03.033 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7000 IOs, 2982612992 bytes 2024-09-17T23:22:03.162 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7100 IOs, 3018692608 bytes 2024-09-17T23:22:03.252 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7200 IOs, 3029341184 bytes 2024-09-17T23:22:03.358 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7300 IOs, 3051504640 bytes 2024-09-17T23:22:03.495 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7400 IOs, 3106207744 bytes 2024-09-17T23:22:03.611 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7500 IOs, 3140262912 bytes 2024-09-17T23:22:03.706 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7600 IOs, 3174242304 bytes 2024-09-17T23:22:03.816 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7700 IOs, 3196894208 bytes 2024-09-17T23:22:03.915 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7800 IOs, 3197513728 bytes 2024-09-17T23:22:04.087 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7900 IOs, 3287097344 bytes 2024-09-17T23:22:04.198 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8000 IOs, 3316802560 bytes 2024-09-17T23:22:04.300 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8100 IOs, 3334133760 bytes 2024-09-17T23:22:04.443 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8200 IOs, 3396466688 bytes 2024-09-17T23:22:04.558 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8300 IOs, 3425495040 bytes 2024-09-17T23:22:04.672 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8400 IOs, 3471632384 bytes 2024-09-17T23:22:04.839 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8500 IOs, 3546032128 bytes 2024-09-17T23:22:04.984 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8600 IOs, 3605497856 bytes 2024-09-17T23:22:05.163 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8700 IOs, 3704560640 bytes 2024-09-17T23:22:05.305 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8800 IOs, 3767248896 bytes 2024-09-17T23:22:05.413 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8900 IOs, 3808798720 bytes 2024-09-17T23:22:05.538 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9000 IOs, 3828127744 bytes 2024-09-17T23:22:05.704 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9100 IOs, 3911990272 bytes 2024-09-17T23:22:05.808 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9200 IOs, 3934551040 bytes 2024-09-17T23:22:05.898 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9300 IOs, 3943472128 bytes 2024-09-17T23:22:06.061 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9400 IOs, 4026499072 bytes 2024-09-17T23:22:06.229 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9500 IOs, 4097769472 bytes 2024-09-17T23:22:06.310 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9600 IOs, 4098378752 bytes 2024-09-17T23:22:06.466 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9700 IOs, 4171502592 bytes 2024-09-17T23:22:06.594 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9800 IOs, 4219158528 bytes 2024-09-17T23:22:06.730 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9900 IOs, 4272524288 bytes 2024-09-17T23:22:06.833 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10000 IOs, 4290708480 bytes 2024-09-17T23:22:06.956 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10100 IOs, 4332478464 bytes 2024-09-17T23:22:07.066 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10200 IOs, 4373278720 bytes 2024-09-17T23:22:07.175 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ error_code=124 2024-09-17T23:22:07.175 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set -e 2024-09-17T23:22:07.175 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' 124 -eq 124 ']' 2024-09-17T23:22:07.175 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:22:07.175 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:22:07.175 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_34 2024-09-17T23:22:07.176 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_image_and_enable_mirror cluster2 mirror image_34 snapshot 128M 2024-09-17T23:22:07.176 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:22:07.176 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:22:07.176 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:22:07.176 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:22:07.176 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_34 2024-09-17T23:22:07.176 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:22:07.177 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=snapshot 2024-09-17T23:22:07.177 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n snapshot ']' 2024-09-17T23:22:07.177 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:22:07.177 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_image cluster2 mirror image_34 128M 2024-09-17T23:22:07.178 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:22:07.178 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:22:07.178 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:22:07.178 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:22:07.178 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_34 2024-09-17T23:22:07.178 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:22:07.178 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local size=128 2024-09-17T23:22:07.178 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n 128M ']' 2024-09-17T23:22:07.178 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ size=128M 2024-09-17T23:22:07.178 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:22:07.178 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_34 2024-09-17T23:22:07.226 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ is_pool_mirror_mode_image mirror 2024-09-17T23:22:07.227 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:22:07.227 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' image = image ']' 2024-09-17T23:22:07.227 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:22:07.227 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ enable_mirror cluster2 mirror image_34 snapshot 2024-09-17T23:22:07.227 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:22:07.227 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:22:07.227 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_34 2024-09-17T23:22:07.227 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=snapshot 2024-09-17T23:22:07.227 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_34 snapshot 2024-09-17T23:22:08.135 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Mirroring enabled 2024-09-17T23:22:08.147 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 info mirror/image_34 2024-09-17T23:22:08.188 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'image_34': 2024-09-17T23:22:08.188 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 128 MiB in 32 objects 2024-09-17T23:22:08.188 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:22:08.188 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 1 2024-09-17T23:22:08.188 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 1427735d8705 2024-09-17T23:22:08.188 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.1427735d8705 2024-09-17T23:22:08.188 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:22:08.188 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-17T23:22:08.188 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:22:08.188 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:22:08.188 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:22:07 2024 2024-09-17T23:22:08.188 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:22:07 2024 2024-09-17T23:22:08.188 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:22:07 2024 2024-09-17T23:22:08.189 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:22:08.189 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:22:08.189 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: 78b7bbf1-8bd9-4fb6-b1bb-58a0b0412b92 2024-09-17T23:22:08.189 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: true 2024-09-17T23:22:08.191 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n '' ']' 2024-09-17T23:22:08.191 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ stress_write_image cluster2 mirror image_34 2024-09-17T23:22:08.191 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:22:08.191 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:22:08.191 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_34 2024-09-17T23:22:08.192 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-17T23:22:08.193 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local duration=6 2024-09-17T23:22:08.193 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set +e 2024-09-17T23:22:08.193 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_34 --debug-rbd=20 --debug-journaler=20 2024-09-17T23:22:08.382 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:100 IOs, 17244160 bytes 2024-09-17T23:22:08.462 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:200 IOs, 17857536 bytes 2024-09-17T23:22:08.562 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:300 IOs, 38248448 bytes 2024-09-17T23:22:08.684 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:400 IOs, 85318656 bytes 2024-09-17T23:22:08.792 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:500 IOs, 112572416 bytes 2024-09-17T23:22:08.886 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:600 IOs, 123706368 bytes 2024-09-17T23:22:08.965 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:700 IOs, 124337152 bytes 2024-09-17T23:22:09.085 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:800 IOs, 162971648 bytes 2024-09-17T23:22:09.190 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:900 IOs, 184966144 bytes 2024-09-17T23:22:09.276 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1000 IOs, 185569280 bytes 2024-09-17T23:22:09.363 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1100 IOs, 209143808 bytes 2024-09-17T23:22:09.512 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1200 IOs, 254863360 bytes 2024-09-17T23:22:09.659 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1300 IOs, 325989376 bytes 2024-09-17T23:22:09.761 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1400 IOs, 348424192 bytes 2024-09-17T23:22:09.886 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1500 IOs, 390677504 bytes 2024-09-17T23:22:10.012 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1600 IOs, 436908032 bytes 2024-09-17T23:22:10.170 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1700 IOs, 509148160 bytes 2024-09-17T23:22:10.296 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1800 IOs, 548335616 bytes 2024-09-17T23:22:10.400 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1900 IOs, 568064000 bytes 2024-09-17T23:22:10.531 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2000 IOs, 620546048 bytes 2024-09-17T23:22:10.620 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2100 IOs, 628690944 bytes 2024-09-17T23:22:10.727 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2200 IOs, 653095936 bytes 2024-09-17T23:22:10.828 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2300 IOs, 675130368 bytes 2024-09-17T23:22:10.982 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2400 IOs, 757668864 bytes 2024-09-17T23:22:11.132 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2500 IOs, 814579712 bytes 2024-09-17T23:22:11.263 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2600 IOs, 863015936 bytes 2024-09-17T23:22:11.382 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2700 IOs, 895264768 bytes 2024-09-17T23:22:11.484 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2800 IOs, 920950784 bytes 2024-09-17T23:22:11.592 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2900 IOs, 947093504 bytes 2024-09-17T23:22:11.702 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3000 IOs, 975953920 bytes 2024-09-17T23:22:11.856 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3100 IOs, 1055441920 bytes 2024-09-17T23:22:11.945 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3200 IOs, 1063547904 bytes 2024-09-17T23:22:12.074 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3300 IOs, 1108573184 bytes 2024-09-17T23:22:12.236 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3400 IOs, 1194992640 bytes 2024-09-17T23:22:12.393 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3500 IOs, 1257901056 bytes 2024-09-17T23:22:12.522 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3600 IOs, 1316032512 bytes 2024-09-17T23:22:12.611 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3700 IOs, 1317729280 bytes 2024-09-17T23:22:12.755 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3800 IOs, 1393982464 bytes 2024-09-17T23:22:12.875 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3900 IOs, 1426565120 bytes 2024-09-17T23:22:12.970 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4000 IOs, 1437240320 bytes 2024-09-17T23:22:13.100 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4100 IOs, 1488295936 bytes 2024-09-17T23:22:13.246 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4200 IOs, 1547321344 bytes 2024-09-17T23:22:13.332 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4300 IOs, 1547957248 bytes 2024-09-17T23:22:13.432 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4400 IOs, 1569938432 bytes 2024-09-17T23:22:13.556 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4500 IOs, 1613106176 bytes 2024-09-17T23:22:13.673 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4600 IOs, 1649625088 bytes 2024-09-17T23:22:13.817 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4700 IOs, 1714539520 bytes 2024-09-17T23:22:13.914 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4800 IOs, 1735854080 bytes 2024-09-17T23:22:14.067 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4900 IOs, 1806444544 bytes 2024-09-17T23:22:14.186 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5000 IOs, 1844559872 bytes 2024-09-17T23:22:14.199 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ error_code=124 2024-09-17T23:22:14.200 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set -e 2024-09-17T23:22:14.200 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' 124 -eq 124 ']' 2024-09-17T23:22:14.200 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:22:14.200 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:22:14.200 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_35 2024-09-17T23:22:14.200 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_image_and_enable_mirror cluster2 mirror image_35 snapshot 128M 2024-09-17T23:22:14.200 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:22:14.200 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:22:14.200 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:22:14.200 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:22:14.201 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_35 2024-09-17T23:22:14.201 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:22:14.201 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=snapshot 2024-09-17T23:22:14.201 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n snapshot ']' 2024-09-17T23:22:14.201 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:22:14.201 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_image cluster2 mirror image_35 128M 2024-09-17T23:22:14.201 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:22:14.201 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:22:14.201 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:22:14.201 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:22:14.201 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_35 2024-09-17T23:22:14.201 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:22:14.201 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local size=128 2024-09-17T23:22:14.201 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n 128M ']' 2024-09-17T23:22:14.201 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ size=128M 2024-09-17T23:22:14.202 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:22:14.202 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_35 2024-09-17T23:22:14.250 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ is_pool_mirror_mode_image mirror 2024-09-17T23:22:14.250 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:22:14.250 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' image = image ']' 2024-09-17T23:22:14.250 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:22:14.250 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ enable_mirror cluster2 mirror image_35 snapshot 2024-09-17T23:22:14.250 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:22:14.250 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:22:14.251 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_35 2024-09-17T23:22:14.251 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=snapshot 2024-09-17T23:22:14.251 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_35 snapshot 2024-09-17T23:22:15.153 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Mirroring enabled 2024-09-17T23:22:15.165 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 info mirror/image_35 2024-09-17T23:22:15.205 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'image_35': 2024-09-17T23:22:15.205 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 128 MiB in 32 objects 2024-09-17T23:22:15.205 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:22:15.205 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 1 2024-09-17T23:22:15.205 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 1433e7c0659c 2024-09-17T23:22:15.205 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.1433e7c0659c 2024-09-17T23:22:15.205 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:22:15.206 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-17T23:22:15.206 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:22:15.206 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:22:15.206 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:22:14 2024 2024-09-17T23:22:15.206 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:22:14 2024 2024-09-17T23:22:15.206 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:22:14 2024 2024-09-17T23:22:15.206 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:22:15.206 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:22:15.206 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: b8f54ad1-6d1a-42f7-821f-2ff1a07ca075 2024-09-17T23:22:15.206 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: true 2024-09-17T23:22:15.209 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n '' ']' 2024-09-17T23:22:15.209 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ stress_write_image cluster2 mirror image_35 2024-09-17T23:22:15.209 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:22:15.209 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:22:15.209 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_35 2024-09-17T23:22:15.209 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-17T23:22:15.212 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local duration=11 2024-09-17T23:22:15.212 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set +e 2024-09-17T23:22:15.212 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_35 --debug-rbd=20 --debug-journaler=20 2024-09-17T23:22:15.412 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:100 IOs, 47108096 bytes 2024-09-17T23:22:15.508 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:200 IOs, 50388992 bytes 2024-09-17T23:22:15.611 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:300 IOs, 74913792 bytes 2024-09-17T23:22:15.753 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:400 IOs, 134352896 bytes 2024-09-17T23:22:15.858 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:500 IOs, 156372992 bytes 2024-09-17T23:22:15.951 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:600 IOs, 170424320 bytes 2024-09-17T23:22:16.053 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:700 IOs, 187577344 bytes 2024-09-17T23:22:16.191 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:800 IOs, 240679936 bytes 2024-09-17T23:22:16.322 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:900 IOs, 288982016 bytes 2024-09-17T23:22:16.416 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1000 IOs, 297930752 bytes 2024-09-17T23:22:16.557 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1100 IOs, 362071040 bytes 2024-09-17T23:22:16.708 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1200 IOs, 437099520 bytes 2024-09-17T23:22:16.867 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1300 IOs, 510527488 bytes 2024-09-17T23:22:17.007 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1400 IOs, 567334912 bytes 2024-09-17T23:22:17.173 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1500 IOs, 665464832 bytes 2024-09-17T23:22:17.346 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1600 IOs, 735923200 bytes 2024-09-17T23:22:17.457 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1700 IOs, 762014720 bytes 2024-09-17T23:22:17.576 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1800 IOs, 800850944 bytes 2024-09-17T23:22:17.662 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1900 IOs, 806474752 bytes 2024-09-17T23:22:17.749 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2000 IOs, 812054528 bytes 2024-09-17T23:22:17.868 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2100 IOs, 846448640 bytes 2024-09-17T23:22:18.012 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2200 IOs, 913466368 bytes 2024-09-17T23:22:18.133 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2300 IOs, 956428288 bytes 2024-09-17T23:22:18.261 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2400 IOs, 988989440 bytes 2024-09-17T23:22:18.417 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2500 IOs, 1056629760 bytes 2024-09-17T23:22:18.527 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2600 IOs, 1084731392 bytes 2024-09-17T23:22:18.638 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2700 IOs, 1114740736 bytes 2024-09-17T23:22:18.761 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2800 IOs, 1155939328 bytes 2024-09-17T23:22:18.871 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2900 IOs, 1184958464 bytes 2024-09-17T23:22:18.982 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3000 IOs, 1211979776 bytes 2024-09-17T23:22:19.125 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3100 IOs, 1272048640 bytes 2024-09-17T23:22:19.231 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3200 IOs, 1307370496 bytes 2024-09-17T23:22:19.340 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3300 IOs, 1319108608 bytes 2024-09-17T23:22:19.462 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3400 IOs, 1369690112 bytes 2024-09-17T23:22:19.577 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3500 IOs, 1392574464 bytes 2024-09-17T23:22:19.725 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3600 IOs, 1454548992 bytes 2024-09-17T23:22:19.814 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3700 IOs, 1472668672 bytes 2024-09-17T23:22:19.945 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3800 IOs, 1507604480 bytes 2024-09-17T23:22:20.055 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3900 IOs, 1570121728 bytes 2024-09-17T23:22:20.260 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4000 IOs, 1657028608 bytes 2024-09-17T23:22:20.382 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4100 IOs, 1699990528 bytes 2024-09-17T23:22:20.575 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4200 IOs, 1804466176 bytes 2024-09-17T23:22:20.689 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4300 IOs, 1835842560 bytes 2024-09-17T23:22:20.791 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4400 IOs, 1855790080 bytes 2024-09-17T23:22:20.889 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4500 IOs, 1875072000 bytes 2024-09-17T23:22:20.981 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4600 IOs, 1886022656 bytes 2024-09-17T23:22:21.136 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4700 IOs, 1950901248 bytes 2024-09-17T23:22:21.315 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4800 IOs, 2046701568 bytes 2024-09-17T23:22:21.427 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4900 IOs, 2072070144 bytes 2024-09-17T23:22:21.539 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5000 IOs, 2101943296 bytes 2024-09-17T23:22:21.679 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5100 IOs, 2157694976 bytes 2024-09-17T23:22:21.792 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5200 IOs, 2192023552 bytes 2024-09-17T23:22:21.893 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5300 IOs, 2207132672 bytes 2024-09-17T23:22:22.024 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5400 IOs, 2252372992 bytes 2024-09-17T23:22:22.129 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5500 IOs, 2273453056 bytes 2024-09-17T23:22:22.255 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5600 IOs, 2318153728 bytes 2024-09-17T23:22:22.419 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5700 IOs, 2395084800 bytes 2024-09-17T23:22:22.531 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5800 IOs, 2422571008 bytes 2024-09-17T23:22:22.662 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5900 IOs, 2474761216 bytes 2024-09-17T23:22:22.767 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6000 IOs, 2494177280 bytes 2024-09-17T23:22:22.874 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6100 IOs, 2517936128 bytes 2024-09-17T23:22:23.074 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6200 IOs, 2632981504 bytes 2024-09-17T23:22:23.177 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6300 IOs, 2676903936 bytes 2024-09-17T23:22:23.284 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6400 IOs, 2677524480 bytes 2024-09-17T23:22:23.456 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6500 IOs, 2768517120 bytes 2024-09-17T23:22:23.567 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6600 IOs, 2809751552 bytes 2024-09-17T23:22:23.747 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6700 IOs, 2900194304 bytes 2024-09-17T23:22:23.890 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6800 IOs, 2960432128 bytes 2024-09-17T23:22:24.020 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6900 IOs, 3009671168 bytes 2024-09-17T23:22:24.175 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7000 IOs, 3081915392 bytes 2024-09-17T23:22:24.297 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7100 IOs, 3125613568 bytes 2024-09-17T23:22:24.418 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7200 IOs, 3153268736 bytes 2024-09-17T23:22:24.564 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7300 IOs, 3219772416 bytes 2024-09-17T23:22:24.701 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7400 IOs, 3268361216 bytes 2024-09-17T23:22:24.803 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7500 IOs, 3294116864 bytes 2024-09-17T23:22:24.904 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7600 IOs, 3299256320 bytes 2024-09-17T23:22:25.016 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7700 IOs, 3329273856 bytes 2024-09-17T23:22:25.136 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7800 IOs, 3369532416 bytes 2024-09-17T23:22:25.252 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7900 IOs, 3405165568 bytes 2024-09-17T23:22:25.382 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8000 IOs, 3449496576 bytes 2024-09-17T23:22:25.515 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8100 IOs, 3498773504 bytes 2024-09-17T23:22:25.635 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8200 IOs, 3529359360 bytes 2024-09-17T23:22:25.789 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8300 IOs, 3602658304 bytes 2024-09-17T23:22:25.918 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8400 IOs, 3652497408 bytes 2024-09-17T23:22:26.144 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8500 IOs, 3802651648 bytes 2024-09-17T23:22:26.217 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ error_code=124 2024-09-17T23:22:26.217 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set -e 2024-09-17T23:22:26.217 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' 124 -eq 124 ']' 2024-09-17T23:22:26.217 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:22:26.217 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:22:26.217 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_36 2024-09-17T23:22:26.217 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_image_and_enable_mirror cluster2 mirror image_36 snapshot 128M 2024-09-17T23:22:26.218 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:22:26.218 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:22:26.218 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:22:26.218 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:22:26.218 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_36 2024-09-17T23:22:26.218 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:22:26.218 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=snapshot 2024-09-17T23:22:26.218 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n snapshot ']' 2024-09-17T23:22:26.218 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:22:26.218 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_image cluster2 mirror image_36 128M 2024-09-17T23:22:26.218 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:22:26.218 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:22:26.218 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:22:26.218 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:22:26.218 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_36 2024-09-17T23:22:26.219 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:22:26.219 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local size=128 2024-09-17T23:22:26.219 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n 128M ']' 2024-09-17T23:22:26.219 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ size=128M 2024-09-17T23:22:26.219 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:22:26.219 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_36 2024-09-17T23:22:26.270 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ is_pool_mirror_mode_image mirror 2024-09-17T23:22:26.271 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:22:26.271 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' image = image ']' 2024-09-17T23:22:26.271 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:22:26.271 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ enable_mirror cluster2 mirror image_36 snapshot 2024-09-17T23:22:26.271 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:22:26.271 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:22:26.271 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_36 2024-09-17T23:22:26.271 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=snapshot 2024-09-17T23:22:26.271 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_36 snapshot 2024-09-17T23:22:27.164 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Mirroring enabled 2024-09-17T23:22:27.174 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 info mirror/image_36 2024-09-17T23:22:27.212 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'image_36': 2024-09-17T23:22:27.212 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 128 MiB in 32 objects 2024-09-17T23:22:27.212 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:22:27.212 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 1 2024-09-17T23:22:27.213 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 143fb24413b0 2024-09-17T23:22:27.213 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.143fb24413b0 2024-09-17T23:22:27.213 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:22:27.213 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-17T23:22:27.213 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:22:27.213 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:22:27.213 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:22:26 2024 2024-09-17T23:22:27.213 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:22:26 2024 2024-09-17T23:22:27.213 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:22:26 2024 2024-09-17T23:22:27.213 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:22:27.213 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:22:27.213 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: 2e12bc1d-2eb3-49b1-b5a5-e5c5013f32d1 2024-09-17T23:22:27.214 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: true 2024-09-17T23:22:27.215 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n '' ']' 2024-09-17T23:22:27.216 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ stress_write_image cluster2 mirror image_36 2024-09-17T23:22:27.216 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:22:27.216 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:22:27.216 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_36 2024-09-17T23:22:27.216 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-17T23:22:27.217 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local duration=13 2024-09-17T23:22:27.217 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set +e 2024-09-17T23:22:27.218 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_36 --debug-rbd=20 --debug-journaler=20 2024-09-17T23:22:27.390 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:100 IOs, 7969792 bytes 2024-09-17T23:22:27.476 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:200 IOs, 10304512 bytes 2024-09-17T23:22:27.599 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:300 IOs, 69087232 bytes 2024-09-17T23:22:27.712 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:400 IOs, 96037888 bytes 2024-09-17T23:22:27.829 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:500 IOs, 150415360 bytes 2024-09-17T23:22:27.938 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:600 IOs, 164898816 bytes 2024-09-17T23:22:28.033 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:700 IOs, 179410944 bytes 2024-09-17T23:22:28.187 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:800 IOs, 254365696 bytes 2024-09-17T23:22:28.308 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:900 IOs, 294055936 bytes 2024-09-17T23:22:28.392 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1000 IOs, 298054656 bytes 2024-09-17T23:22:28.505 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1100 IOs, 348285952 bytes 2024-09-17T23:22:28.625 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1200 IOs, 370866176 bytes 2024-09-17T23:22:28.736 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1300 IOs, 397757440 bytes 2024-09-17T23:22:28.872 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1400 IOs, 465433600 bytes 2024-09-17T23:22:29.019 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1500 IOs, 521901056 bytes 2024-09-17T23:22:29.126 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1600 IOs, 552701952 bytes 2024-09-17T23:22:29.212 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1700 IOs, 553323520 bytes 2024-09-17T23:22:29.317 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1800 IOs, 573107200 bytes 2024-09-17T23:22:29.431 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1900 IOs, 608446464 bytes 2024-09-17T23:22:29.558 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2000 IOs, 655915008 bytes 2024-09-17T23:22:29.674 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2100 IOs, 698819584 bytes 2024-09-17T23:22:29.814 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2200 IOs, 740816896 bytes 2024-09-17T23:22:29.954 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2300 IOs, 798931968 bytes 2024-09-17T23:22:30.086 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2400 IOs, 851204096 bytes 2024-09-17T23:22:30.237 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2500 IOs, 929989632 bytes 2024-09-17T23:22:30.387 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2600 IOs, 994824192 bytes 2024-09-17T23:22:30.505 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2700 IOs, 1028924416 bytes 2024-09-17T23:22:30.644 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2800 IOs, 1084582912 bytes 2024-09-17T23:22:30.756 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2900 IOs, 1119476736 bytes 2024-09-17T23:22:30.838 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3000 IOs, 1120082944 bytes 2024-09-17T23:22:30.966 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3100 IOs, 1189608448 bytes 2024-09-17T23:22:31.133 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3200 IOs, 1251847168 bytes 2024-09-17T23:22:31.227 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3300 IOs, 1263711232 bytes 2024-09-17T23:22:31.324 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3400 IOs, 1279964160 bytes 2024-09-17T23:22:31.456 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3500 IOs, 1329318912 bytes 2024-09-17T23:22:31.587 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3600 IOs, 1382415360 bytes 2024-09-17T23:22:31.679 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3700 IOs, 1407245312 bytes 2024-09-17T23:22:31.801 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3800 IOs, 1430122496 bytes 2024-09-17T23:22:31.900 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3900 IOs, 1448799232 bytes 2024-09-17T23:22:32.030 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4000 IOs, 1498114048 bytes 2024-09-17T23:22:32.144 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4100 IOs, 1530852352 bytes 2024-09-17T23:22:32.268 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4200 IOs, 1571505152 bytes 2024-09-17T23:22:32.409 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4300 IOs, 1627602944 bytes 2024-09-17T23:22:32.520 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4400 IOs, 1695178752 bytes 2024-09-17T23:22:32.673 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4500 IOs, 1752335360 bytes 2024-09-17T23:22:32.770 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4600 IOs, 1770427392 bytes 2024-09-17T23:22:32.958 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4700 IOs, 1868284928 bytes 2024-09-17T23:22:33.059 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4800 IOs, 1885526016 bytes 2024-09-17T23:22:33.143 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4900 IOs, 1886130176 bytes 2024-09-17T23:22:33.254 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5000 IOs, 1926397952 bytes 2024-09-17T23:22:33.381 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5100 IOs, 1954263040 bytes 2024-09-17T23:22:33.513 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5200 IOs, 2002001920 bytes 2024-09-17T23:22:33.671 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5300 IOs, 2070406144 bytes 2024-09-17T23:22:33.799 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5400 IOs, 2114320384 bytes 2024-09-17T23:22:33.923 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5500 IOs, 2155069440 bytes 2024-09-17T23:22:34.067 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5600 IOs, 2218658816 bytes 2024-09-17T23:22:34.153 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5700 IOs, 2219280384 bytes 2024-09-17T23:22:34.280 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5800 IOs, 2262779904 bytes 2024-09-17T23:22:34.373 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5900 IOs, 2273588224 bytes 2024-09-17T23:22:34.473 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6000 IOs, 2289774592 bytes 2024-09-17T23:22:34.565 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6100 IOs, 2306641920 bytes 2024-09-17T23:22:34.739 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6200 IOs, 2387749888 bytes 2024-09-17T23:22:34.840 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6300 IOs, 2406995968 bytes 2024-09-17T23:22:34.934 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6400 IOs, 2418064384 bytes 2024-09-17T23:22:35.068 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6500 IOs, 2464992256 bytes 2024-09-17T23:22:35.194 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6600 IOs, 2508968960 bytes 2024-09-17T23:22:35.290 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6700 IOs, 2519715840 bytes 2024-09-17T23:22:35.381 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6800 IOs, 2526927872 bytes 2024-09-17T23:22:35.474 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6900 IOs, 2536581120 bytes 2024-09-17T23:22:35.590 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7000 IOs, 2563636224 bytes 2024-09-17T23:22:35.718 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7100 IOs, 2610594816 bytes 2024-09-17T23:22:35.838 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7200 IOs, 2645577728 bytes 2024-09-17T23:22:35.969 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7300 IOs, 2696219648 bytes 2024-09-17T23:22:36.079 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7400 IOs, 2715424768 bytes 2024-09-17T23:22:36.204 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7500 IOs, 2761742336 bytes 2024-09-17T23:22:36.370 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7600 IOs, 2844182528 bytes 2024-09-17T23:22:36.474 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7700 IOs, 2863331328 bytes 2024-09-17T23:22:36.607 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7800 IOs, 2905910272 bytes 2024-09-17T23:22:36.726 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7900 IOs, 2938924032 bytes 2024-09-17T23:22:36.851 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8000 IOs, 2995533824 bytes 2024-09-17T23:22:36.983 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8100 IOs, 3032786944 bytes 2024-09-17T23:22:37.096 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8200 IOs, 3059353600 bytes 2024-09-17T23:22:37.219 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8300 IOs, 3099066368 bytes 2024-09-17T23:22:37.339 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8400 IOs, 3135392768 bytes 2024-09-17T23:22:37.466 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8500 IOs, 3183286272 bytes 2024-09-17T23:22:37.645 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8600 IOs, 3278967808 bytes 2024-09-17T23:22:37.743 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8700 IOs, 3299322880 bytes 2024-09-17T23:22:37.880 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8800 IOs, 3346934784 bytes 2024-09-17T23:22:37.910 INFO:tasks.ceph.cluster2.osd.1.smithi038.stderr:2024-09-17T23:22:37.908+0000 7f054ba99640 -1 reset not still connected to 0x55df1f511c70 2024-09-17T23:22:37.976 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8900 IOs, 3360494592 bytes 2024-09-17T23:22:38.086 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9000 IOs, 3393862656 bytes 2024-09-17T23:22:38.208 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9100 IOs, 3435575296 bytes 2024-09-17T23:22:38.304 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9200 IOs, 3436186624 bytes 2024-09-17T23:22:38.397 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9300 IOs, 3449400320 bytes 2024-09-17T23:22:38.575 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9400 IOs, 3538799616 bytes 2024-09-17T23:22:38.687 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9500 IOs, 3574309888 bytes 2024-09-17T23:22:38.891 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9600 IOs, 3694213120 bytes 2024-09-17T23:22:39.010 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9700 IOs, 3725907968 bytes 2024-09-17T23:22:39.138 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9800 IOs, 3763737600 bytes 2024-09-17T23:22:39.248 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9900 IOs, 3792454656 bytes 2024-09-17T23:22:39.382 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10000 IOs, 3844963328 bytes 2024-09-17T23:22:39.511 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10100 IOs, 3895269376 bytes 2024-09-17T23:22:39.617 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10200 IOs, 3915970560 bytes 2024-09-17T23:22:39.726 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10300 IOs, 3941565440 bytes 2024-09-17T23:22:39.828 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10400 IOs, 3957438464 bytes 2024-09-17T23:22:39.972 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10500 IOs, 4022403072 bytes 2024-09-17T23:22:40.082 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10600 IOs, 4052892672 bytes 2024-09-17T23:22:40.225 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ error_code=124 2024-09-17T23:22:40.225 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set -e 2024-09-17T23:22:40.225 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' 124 -eq 124 ']' 2024-09-17T23:22:40.226 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:22:40.226 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:22:40.226 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_37 2024-09-17T23:22:40.226 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_image_and_enable_mirror cluster2 mirror image_37 snapshot 128M 2024-09-17T23:22:40.226 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:22:40.226 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:22:40.226 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:22:40.226 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:22:40.226 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_37 2024-09-17T23:22:40.226 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:22:40.226 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=snapshot 2024-09-17T23:22:40.227 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n snapshot ']' 2024-09-17T23:22:40.227 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:22:40.227 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_image cluster2 mirror image_37 128M 2024-09-17T23:22:40.227 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:22:40.227 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:22:40.227 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:22:40.227 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:22:40.227 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_37 2024-09-17T23:22:40.227 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:22:40.227 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local size=128 2024-09-17T23:22:40.227 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n 128M ']' 2024-09-17T23:22:40.227 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ size=128M 2024-09-17T23:22:40.227 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:22:40.227 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_37 2024-09-17T23:22:40.277 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ is_pool_mirror_mode_image mirror 2024-09-17T23:22:40.277 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:22:40.278 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' image = image ']' 2024-09-17T23:22:40.278 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:22:40.278 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ enable_mirror cluster2 mirror image_37 snapshot 2024-09-17T23:22:40.278 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:22:40.278 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:22:40.278 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_37 2024-09-17T23:22:40.278 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=snapshot 2024-09-17T23:22:40.278 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_37 snapshot 2024-09-17T23:22:41.195 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Mirroring enabled 2024-09-17T23:22:41.206 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 info mirror/image_37 2024-09-17T23:22:41.246 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'image_37': 2024-09-17T23:22:41.246 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 128 MiB in 32 objects 2024-09-17T23:22:41.246 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:22:41.247 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 1 2024-09-17T23:22:41.247 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 144b1be5bc2b 2024-09-17T23:22:41.247 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.144b1be5bc2b 2024-09-17T23:22:41.247 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:22:41.247 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-17T23:22:41.247 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:22:41.247 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:22:41.247 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:22:40 2024 2024-09-17T23:22:41.247 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:22:40 2024 2024-09-17T23:22:41.247 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:22:40 2024 2024-09-17T23:22:41.247 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:22:41.247 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:22:41.247 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: ccc8bbca-2b19-4cff-8cb4-7770b06359dc 2024-09-17T23:22:41.247 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: true 2024-09-17T23:22:41.250 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n '' ']' 2024-09-17T23:22:41.250 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ stress_write_image cluster2 mirror image_37 2024-09-17T23:22:41.250 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:22:41.250 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:22:41.250 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_37 2024-09-17T23:22:41.250 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-17T23:22:41.252 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local duration=9 2024-09-17T23:22:41.252 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set +e 2024-09-17T23:22:41.252 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_37 --debug-rbd=20 --debug-journaler=20 2024-09-17T23:22:41.420 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:100 IOs, 19942400 bytes 2024-09-17T23:22:41.592 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:200 IOs, 94914560 bytes 2024-09-17T23:22:41.700 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:300 IOs, 131428352 bytes 2024-09-17T23:22:41.827 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:400 IOs, 161733632 bytes 2024-09-17T23:22:41.951 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:500 IOs, 206530560 bytes 2024-09-17T23:22:42.038 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:600 IOs, 212500480 bytes 2024-09-17T23:22:42.165 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:700 IOs, 255505408 bytes 2024-09-17T23:22:42.306 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:800 IOs, 324546560 bytes 2024-09-17T23:22:42.414 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:900 IOs, 341226496 bytes 2024-09-17T23:22:42.499 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1000 IOs, 341857280 bytes 2024-09-17T23:22:42.651 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1100 IOs, 408537088 bytes 2024-09-17T23:22:42.765 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1200 IOs, 437338112 bytes 2024-09-17T23:22:42.945 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1300 IOs, 539463680 bytes 2024-09-17T23:22:43.042 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1400 IOs, 553331712 bytes 2024-09-17T23:22:43.141 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1500 IOs, 570789888 bytes 2024-09-17T23:22:43.262 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1600 IOs, 603610112 bytes 2024-09-17T23:22:43.409 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1700 IOs, 680667136 bytes 2024-09-17T23:22:43.531 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1800 IOs, 704915456 bytes 2024-09-17T23:22:43.623 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1900 IOs, 726279168 bytes 2024-09-17T23:22:43.729 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2000 IOs, 741700608 bytes 2024-09-17T23:22:43.813 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2100 IOs, 743071744 bytes 2024-09-17T23:22:43.829 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.8313778175825861) > weight (0.25) 2024-09-17T23:22:43.829 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2024-09-17T23:22:43.829 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2024-09-17T23:22:43.830 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2024-09-17T23:22:43.831 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi038.stderr:2024-09-17T23:22:43.828+0000 7f95a64ea640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 46943) UID: 1000 2024-09-17T23:22:43.988 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2200 IOs, 828492800 bytes 2024-09-17T23:22:44.105 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2300 IOs, 862653440 bytes 2024-09-17T23:22:44.228 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2400 IOs, 902913024 bytes 2024-09-17T23:22:44.373 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2500 IOs, 963560448 bytes 2024-09-17T23:22:44.518 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2600 IOs, 1029278720 bytes 2024-09-17T23:22:44.625 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2700 IOs, 1048541184 bytes 2024-09-17T23:22:44.802 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2800 IOs, 1137921024 bytes 2024-09-17T23:22:44.924 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2900 IOs, 1172131840 bytes 2024-09-17T23:22:45.074 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3000 IOs, 1238421504 bytes 2024-09-17T23:22:45.235 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3100 IOs, 1321233408 bytes 2024-09-17T23:22:45.345 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3200 IOs, 1386151936 bytes 2024-09-17T23:22:45.500 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3300 IOs, 1427348480 bytes 2024-09-17T23:22:45.598 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3400 IOs, 1447021568 bytes 2024-09-17T23:22:45.687 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3500 IOs, 1453071360 bytes 2024-09-17T23:22:45.798 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3600 IOs, 1482299392 bytes 2024-09-17T23:22:45.908 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3700 IOs, 1506667520 bytes 2024-09-17T23:22:46.037 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3800 IOs, 1548876800 bytes 2024-09-17T23:22:46.166 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3900 IOs, 1606167552 bytes 2024-09-17T23:22:46.259 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4000 IOs, 1607446528 bytes 2024-09-17T23:22:46.390 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4100 IOs, 1662775296 bytes 2024-09-17T23:22:46.515 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4200 IOs, 1701603328 bytes 2024-09-17T23:22:46.615 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4300 IOs, 1718115328 bytes 2024-09-17T23:22:46.707 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4400 IOs, 1723600896 bytes 2024-09-17T23:22:46.843 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4500 IOs, 1772266496 bytes 2024-09-17T23:22:46.946 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4600 IOs, 1805121536 bytes 2024-09-17T23:22:47.090 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4700 IOs, 1874797568 bytes 2024-09-17T23:22:47.233 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4800 IOs, 1910146048 bytes 2024-09-17T23:22:47.380 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4900 IOs, 1968590848 bytes 2024-09-17T23:22:47.462 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5000 IOs, 1975187456 bytes 2024-09-17T23:22:47.556 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5100 IOs, 1978483712 bytes 2024-09-17T23:22:47.679 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5200 IOs, 2018256896 bytes 2024-09-17T23:22:47.808 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5300 IOs, 2058800128 bytes 2024-09-17T23:22:47.899 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5400 IOs, 2072271872 bytes 2024-09-17T23:22:48.014 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5500 IOs, 2092898304 bytes 2024-09-17T23:22:48.110 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5600 IOs, 2103394304 bytes 2024-09-17T23:22:48.235 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5700 IOs, 2150006784 bytes 2024-09-17T23:22:48.322 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5800 IOs, 2150624256 bytes 2024-09-17T23:22:48.460 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5900 IOs, 2207472640 bytes 2024-09-17T23:22:48.589 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6000 IOs, 2260219904 bytes 2024-09-17T23:22:48.675 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6100 IOs, 2260828160 bytes 2024-09-17T23:22:48.815 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6200 IOs, 2315243520 bytes 2024-09-17T23:22:48.944 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6300 IOs, 2360091648 bytes 2024-09-17T23:22:49.028 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6400 IOs, 2360681472 bytes 2024-09-17T23:22:49.162 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6500 IOs, 2416054272 bytes 2024-09-17T23:22:49.294 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6600 IOs, 2480445440 bytes 2024-09-17T23:22:49.419 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6700 IOs, 2507610112 bytes 2024-09-17T23:22:49.616 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6800 IOs, 2621552640 bytes 2024-09-17T23:22:49.781 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6900 IOs, 2714440704 bytes 2024-09-17T23:22:49.831 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2024-09-17T23:22:49.831 INFO:teuthology.orchestra.run:waiting for 600 2024-09-17T23:22:49.831 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2024-09-17T23:22:49.831 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2024-09-17T23:22:49.832 DEBUG:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2024-09-17T23:22:49.835 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2024-09-17T23:22:49.835 INFO:tasks.rbd_mirror_thrash:waiting for 88 secs before thrashing 2024-09-17T23:22:49.936 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7000 IOs, 2767167488 bytes 2024-09-17T23:22:50.049 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7100 IOs, 2781770752 bytes 2024-09-17T23:22:50.165 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7200 IOs, 2801656832 bytes 2024-09-17T23:22:50.258 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ error_code=124 2024-09-17T23:22:50.258 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set -e 2024-09-17T23:22:50.258 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' 124 -eq 124 ']' 2024-09-17T23:22:50.258 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:22:50.258 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:22:50.258 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_38 2024-09-17T23:22:50.258 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_image_and_enable_mirror cluster2 mirror image_38 snapshot 128M 2024-09-17T23:22:50.258 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:22:50.259 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:22:50.259 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:22:50.259 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:22:50.259 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_38 2024-09-17T23:22:50.259 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:22:50.259 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=snapshot 2024-09-17T23:22:50.259 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n snapshot ']' 2024-09-17T23:22:50.259 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:22:50.260 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_image cluster2 mirror image_38 128M 2024-09-17T23:22:50.260 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:22:50.260 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:22:50.260 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:22:50.260 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:22:50.260 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_38 2024-09-17T23:22:50.260 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:22:50.260 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local size=128 2024-09-17T23:22:50.260 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n 128M ']' 2024-09-17T23:22:50.260 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ size=128M 2024-09-17T23:22:50.260 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:22:50.261 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_38 2024-09-17T23:22:50.309 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ is_pool_mirror_mode_image mirror 2024-09-17T23:22:50.309 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:22:50.310 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' image = image ']' 2024-09-17T23:22:50.310 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:22:50.310 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ enable_mirror cluster2 mirror image_38 snapshot 2024-09-17T23:22:50.310 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:22:50.310 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:22:50.310 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_38 2024-09-17T23:22:50.310 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=snapshot 2024-09-17T23:22:50.310 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_38 snapshot 2024-09-17T23:22:51.260 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Mirroring enabled 2024-09-17T23:22:51.272 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 info mirror/image_38 2024-09-17T23:22:51.313 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'image_38': 2024-09-17T23:22:51.313 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 128 MiB in 32 objects 2024-09-17T23:22:51.313 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:22:51.313 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 1 2024-09-17T23:22:51.313 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 145c9df77a60 2024-09-17T23:22:51.313 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.145c9df77a60 2024-09-17T23:22:51.313 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:22:51.314 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-17T23:22:51.314 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:22:51.314 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:22:51.314 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:22:50 2024 2024-09-17T23:22:51.314 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:22:50 2024 2024-09-17T23:22:51.314 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:22:50 2024 2024-09-17T23:22:51.314 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:22:51.314 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:22:51.314 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: 37d0dace-6324-49a7-9dcc-0827dcbac606 2024-09-17T23:22:51.314 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: true 2024-09-17T23:22:51.317 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n '' ']' 2024-09-17T23:22:51.317 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ stress_write_image cluster2 mirror image_38 2024-09-17T23:22:51.317 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:22:51.317 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:22:51.317 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_38 2024-09-17T23:22:51.317 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-17T23:22:51.319 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local duration=7 2024-09-17T23:22:51.319 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set +e 2024-09-17T23:22:51.319 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_38 --debug-rbd=20 --debug-journaler=20 2024-09-17T23:22:51.524 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:100 IOs, 31768576 bytes 2024-09-17T23:22:51.645 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:200 IOs, 69323776 bytes 2024-09-17T23:22:51.750 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:300 IOs, 96043008 bytes 2024-09-17T23:22:51.885 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:400 IOs, 154468352 bytes 2024-09-17T23:22:52.065 INFO:tasks.ceph.cluster2.osd.1.smithi038.stderr:2024-09-17T23:22:52.063+0000 7f054ba99640 -1 reset not still connected to 0x55df27f98820 2024-09-17T23:22:52.065 INFO:tasks.ceph.cluster2.osd.1.smithi038.stderr:2024-09-17T23:22:52.063+0000 7f054ba99640 -1 reset not still connected to 0x55df29801860 2024-09-17T23:22:52.066 INFO:tasks.ceph.cluster2.osd.1.smithi038.stderr:2024-09-17T23:22:52.063+0000 7f054ba99640 -1 reset not still connected to 0x55df348b8a90 2024-09-17T23:22:52.075 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:500 IOs, 264275968 bytes 2024-09-17T23:22:52.193 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:600 IOs, 294458368 bytes 2024-09-17T23:22:52.377 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:700 IOs, 401461248 bytes 2024-09-17T23:22:52.477 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:800 IOs, 420032512 bytes 2024-09-17T23:22:52.620 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:900 IOs, 480902144 bytes 2024-09-17T23:22:52.729 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1000 IOs, 510779392 bytes 2024-09-17T23:22:52.872 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1100 IOs, 583908352 bytes 2024-09-17T23:22:53.020 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1200 IOs, 632825856 bytes 2024-09-17T23:22:53.139 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1300 IOs, 666989568 bytes 2024-09-17T23:22:53.270 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1400 IOs, 735369216 bytes 2024-09-17T23:22:53.414 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1500 IOs, 771628032 bytes 2024-09-17T23:22:53.550 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1600 IOs, 831241216 bytes 2024-09-17T23:22:53.706 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1700 IOs, 901037056 bytes 2024-09-17T23:22:53.851 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1800 IOs, 963751936 bytes 2024-09-17T23:22:53.957 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1900 IOs, 983930880 bytes 2024-09-17T23:22:54.107 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2000 IOs, 1063938048 bytes 2024-09-17T23:22:54.229 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2100 IOs, 1082241024 bytes 2024-09-17T23:22:54.361 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2200 IOs, 1130416128 bytes 2024-09-17T23:22:54.529 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2300 IOs, 1226933248 bytes 2024-09-17T23:22:54.640 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2400 IOs, 1250400256 bytes 2024-09-17T23:22:54.756 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2500 IOs, 1281689600 bytes 2024-09-17T23:22:54.861 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2600 IOs, 1303060480 bytes 2024-09-17T23:22:54.987 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2700 IOs, 1346087936 bytes 2024-09-17T23:22:55.085 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2800 IOs, 1360392192 bytes 2024-09-17T23:22:55.273 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2900 IOs, 1474438144 bytes 2024-09-17T23:22:55.419 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3000 IOs, 1537142784 bytes 2024-09-17T23:22:55.529 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3100 IOs, 1567648768 bytes 2024-09-17T23:22:55.631 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3200 IOs, 1586795520 bytes 2024-09-17T23:22:55.769 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3300 IOs, 1639875584 bytes 2024-09-17T23:22:55.856 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3400 IOs, 1643713536 bytes 2024-09-17T23:22:55.952 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3500 IOs, 1659196416 bytes 2024-09-17T23:22:56.071 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3600 IOs, 1685097472 bytes 2024-09-17T23:22:56.347 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3700 IOs, 1786564608 bytes 2024-09-17T23:22:56.455 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3800 IOs, 1829916672 bytes 2024-09-17T23:22:56.651 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3900 IOs, 1931412480 bytes 2024-09-17T23:22:56.817 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4000 IOs, 2011658240 bytes 2024-09-17T23:22:56.951 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4100 IOs, 2048001024 bytes 2024-09-17T23:22:57.070 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4200 IOs, 2081170432 bytes 2024-09-17T23:22:57.156 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4300 IOs, 2102439936 bytes 2024-09-17T23:22:57.259 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4400 IOs, 2105573376 bytes 2024-09-17T23:22:57.344 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4500 IOs, 2108488704 bytes 2024-09-17T23:22:57.461 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4600 IOs, 2139796480 bytes 2024-09-17T23:22:57.561 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4700 IOs, 2156973056 bytes 2024-09-17T23:22:57.676 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4800 IOs, 2184426496 bytes 2024-09-17T23:22:57.798 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4900 IOs, 2221928448 bytes 2024-09-17T23:22:57.915 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5000 IOs, 2258931712 bytes 2024-09-17T23:22:58.040 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5100 IOs, 2297134080 bytes 2024-09-17T23:22:58.177 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5200 IOs, 2353524736 bytes 2024-09-17T23:22:58.325 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ error_code=124 2024-09-17T23:22:58.326 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set -e 2024-09-17T23:22:58.326 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' 124 -eq 124 ']' 2024-09-17T23:22:58.326 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:22:58.326 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:22:58.326 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_39 2024-09-17T23:22:58.326 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_image_and_enable_mirror cluster2 mirror image_39 snapshot 128M 2024-09-17T23:22:58.326 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:22:58.326 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:22:58.326 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:22:58.326 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:22:58.326 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_39 2024-09-17T23:22:58.326 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:22:58.327 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=snapshot 2024-09-17T23:22:58.327 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n snapshot ']' 2024-09-17T23:22:58.328 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:22:58.328 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_image cluster2 mirror image_39 128M 2024-09-17T23:22:58.328 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:22:58.328 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:22:58.328 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:22:58.328 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:22:58.328 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_39 2024-09-17T23:22:58.328 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:22:58.328 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local size=128 2024-09-17T23:22:58.328 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n 128M ']' 2024-09-17T23:22:58.328 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ size=128M 2024-09-17T23:22:58.328 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:22:58.328 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_39 2024-09-17T23:22:58.380 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ is_pool_mirror_mode_image mirror 2024-09-17T23:22:58.381 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:22:58.381 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' image = image ']' 2024-09-17T23:22:58.381 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:22:58.381 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ enable_mirror cluster2 mirror image_39 snapshot 2024-09-17T23:22:58.381 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:22:58.381 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:22:58.381 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_39 2024-09-17T23:22:58.381 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=snapshot 2024-09-17T23:22:58.381 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_39 snapshot 2024-09-17T23:22:59.229 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Mirroring enabled 2024-09-17T23:22:59.240 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 info mirror/image_39 2024-09-17T23:22:59.280 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'image_39': 2024-09-17T23:22:59.281 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 128 MiB in 32 objects 2024-09-17T23:22:59.281 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:22:59.281 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 1 2024-09-17T23:22:59.281 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 14685bcc2702 2024-09-17T23:22:59.281 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.14685bcc2702 2024-09-17T23:22:59.281 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:22:59.281 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-17T23:22:59.281 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:22:59.281 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:22:59.281 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:22:58 2024 2024-09-17T23:22:59.281 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:22:58 2024 2024-09-17T23:22:59.281 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:22:58 2024 2024-09-17T23:22:59.281 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:22:59.282 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:22:59.282 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: a0492a7c-0945-424f-9742-2a83199e6bda 2024-09-17T23:22:59.282 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: true 2024-09-17T23:22:59.284 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n '' ']' 2024-09-17T23:22:59.284 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ stress_write_image cluster2 mirror image_39 2024-09-17T23:22:59.284 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:22:59.284 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:22:59.284 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_39 2024-09-17T23:22:59.284 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-17T23:22:59.286 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local duration=13 2024-09-17T23:22:59.286 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set +e 2024-09-17T23:22:59.286 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_39 --debug-rbd=20 --debug-journaler=20 2024-09-17T23:22:59.479 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:100 IOs, 33046528 bytes 2024-09-17T23:22:59.579 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:200 IOs, 44059648 bytes 2024-09-17T23:22:59.743 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:300 IOs, 130605056 bytes 2024-09-17T23:22:59.837 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:400 IOs, 138769408 bytes 2024-09-17T23:22:59.951 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:500 IOs, 184133632 bytes 2024-09-17T23:23:00.081 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:600 IOs, 223256576 bytes 2024-09-17T23:23:00.176 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:700 IOs, 238167040 bytes 2024-09-17T23:23:00.354 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:800 IOs, 333274112 bytes 2024-09-17T23:23:00.460 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:900 IOs, 357395456 bytes 2024-09-17T23:23:00.590 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1000 IOs, 408100864 bytes 2024-09-17T23:23:00.705 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1100 IOs, 451696640 bytes 2024-09-17T23:23:00.823 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1200 IOs, 482476032 bytes 2024-09-17T23:23:00.983 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1300 IOs, 562340864 bytes 2024-09-17T23:23:01.145 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1400 IOs, 661621760 bytes 2024-09-17T23:23:01.274 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1500 IOs, 686614528 bytes 2024-09-17T23:23:01.393 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1600 IOs, 732494848 bytes 2024-09-17T23:23:01.526 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1700 IOs, 773242880 bytes 2024-09-17T23:23:01.638 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1800 IOs, 806340608 bytes 2024-09-17T23:23:01.723 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1900 IOs, 811087872 bytes 2024-09-17T23:23:01.838 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2000 IOs, 844481536 bytes 2024-09-17T23:23:01.945 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2100 IOs, 873299968 bytes 2024-09-17T23:23:02.091 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2200 IOs, 939253760 bytes 2024-09-17T23:23:02.221 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2300 IOs, 983122944 bytes 2024-09-17T23:23:02.346 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2400 IOs, 1039016960 bytes 2024-09-17T23:23:02.453 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2500 IOs, 1052512256 bytes 2024-09-17T23:23:02.576 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2600 IOs, 1096311808 bytes 2024-09-17T23:23:02.697 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2700 IOs, 1134823424 bytes 2024-09-17T23:23:02.834 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2800 IOs, 1193003008 bytes 2024-09-17T23:23:02.953 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2900 IOs, 1232764928 bytes 2024-09-17T23:23:03.059 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3000 IOs, 1259271168 bytes 2024-09-17T23:23:03.144 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3100 IOs, 1259902976 bytes 2024-09-17T23:23:03.292 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3200 IOs, 1325104128 bytes 2024-09-17T23:23:03.399 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3300 IOs, 1352626176 bytes 2024-09-17T23:23:03.521 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3400 IOs, 1393745920 bytes 2024-09-17T23:23:03.638 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3500 IOs, 1428837376 bytes 2024-09-17T23:23:03.728 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3600 IOs, 1447572480 bytes 2024-09-17T23:23:03.894 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3700 IOs, 1540645888 bytes 2024-09-17T23:23:04.012 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3800 IOs, 1563880448 bytes 2024-09-17T23:23:04.134 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3900 IOs, 1596919808 bytes 2024-09-17T23:23:04.285 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4000 IOs, 1667743744 bytes 2024-09-17T23:23:04.438 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4100 IOs, 1739021312 bytes 2024-09-17T23:23:04.523 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4200 IOs, 1739650048 bytes 2024-09-17T23:23:04.662 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4300 IOs, 1800072192 bytes 2024-09-17T23:23:04.810 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4400 IOs, 1870703616 bytes 2024-09-17T23:23:04.914 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4500 IOs, 1893706752 bytes 2024-09-17T23:23:05.249 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4600 IOs, 1973192704 bytes 2024-09-17T23:23:05.353 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4700 IOs, 2015263744 bytes 2024-09-17T23:23:05.476 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4800 IOs, 2041704448 bytes 2024-09-17T23:23:05.618 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4900 IOs, 2105403392 bytes 2024-09-17T23:23:05.724 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5000 IOs, 2128082944 bytes 2024-09-17T23:23:05.841 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5100 IOs, 2164426752 bytes 2024-09-17T23:23:06.004 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5200 IOs, 2246348800 bytes 2024-09-17T23:23:06.118 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5300 IOs, 2274004992 bytes 2024-09-17T23:23:06.271 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5400 IOs, 2337264640 bytes 2024-09-17T23:23:06.422 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5500 IOs, 2410742784 bytes 2024-09-17T23:23:06.548 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5600 IOs, 2464312320 bytes 2024-09-17T23:23:06.696 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5700 IOs, 2538148864 bytes 2024-09-17T23:23:06.804 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5800 IOs, 2560840704 bytes 2024-09-17T23:23:06.921 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5900 IOs, 2593413120 bytes 2024-09-17T23:23:07.027 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6000 IOs, 2617080832 bytes 2024-09-17T23:23:07.140 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6100 IOs, 2660619264 bytes 2024-09-17T23:23:07.316 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6200 IOs, 2741238784 bytes 2024-09-17T23:23:07.436 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6300 IOs, 2781072384 bytes 2024-09-17T23:23:07.579 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6400 IOs, 2849347584 bytes 2024-09-17T23:23:07.686 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6500 IOs, 2876539904 bytes 2024-09-17T23:23:07.799 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6600 IOs, 2910470144 bytes 2024-09-17T23:23:07.916 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6700 IOs, 2947382272 bytes 2024-09-17T23:23:08.021 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6800 IOs, 2974096384 bytes 2024-09-17T23:23:08.136 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6900 IOs, 3006209024 bytes 2024-09-17T23:23:08.247 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7000 IOs, 3030683648 bytes 2024-09-17T23:23:08.359 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7100 IOs, 3061604352 bytes 2024-09-17T23:23:08.465 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7200 IOs, 3086916608 bytes 2024-09-17T23:23:08.599 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7300 IOs, 3132730368 bytes 2024-09-17T23:23:08.765 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7400 IOs, 3217466368 bytes 2024-09-17T23:23:08.848 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7500 IOs, 3218074624 bytes 2024-09-17T23:23:08.996 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7600 IOs, 3280370688 bytes 2024-09-17T23:23:09.106 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7700 IOs, 3303946240 bytes 2024-09-17T23:23:09.192 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7800 IOs, 3322074112 bytes 2024-09-17T23:23:09.299 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7900 IOs, 3330935808 bytes 2024-09-17T23:23:09.406 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8000 IOs, 3352437760 bytes 2024-09-17T23:23:09.503 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8100 IOs, 3382447104 bytes 2024-09-17T23:23:09.612 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8200 IOs, 3408001024 bytes 2024-09-17T23:23:09.714 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8300 IOs, 3415810048 bytes 2024-09-17T23:23:09.805 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8400 IOs, 3422691328 bytes 2024-09-17T23:23:09.979 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8500 IOs, 3513054208 bytes 2024-09-17T23:23:10.107 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8600 IOs, 3562260480 bytes 2024-09-17T23:23:10.203 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8700 IOs, 3567812608 bytes 2024-09-17T23:23:10.326 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8800 IOs, 3609185280 bytes 2024-09-17T23:23:10.463 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8900 IOs, 3665632256 bytes 2024-09-17T23:23:10.597 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9000 IOs, 3723975680 bytes 2024-09-17T23:23:10.717 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9100 IOs, 3762188288 bytes 2024-09-17T23:23:10.838 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9200 IOs, 3809577984 bytes 2024-09-17T23:23:10.969 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9300 IOs, 3858134016 bytes 2024-09-17T23:23:11.084 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9400 IOs, 3887287296 bytes 2024-09-17T23:23:11.269 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9500 IOs, 3988376576 bytes 2024-09-17T23:23:11.366 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9600 IOs, 4004862976 bytes 2024-09-17T23:23:11.488 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9700 IOs, 4040096768 bytes 2024-09-17T23:23:11.610 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9800 IOs, 4081377280 bytes 2024-09-17T23:23:11.697 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9900 IOs, 4088348672 bytes 2024-09-17T23:23:11.814 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10000 IOs, 4131347456 bytes 2024-09-17T23:23:11.947 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10100 IOs, 4179154944 bytes 2024-09-17T23:23:12.084 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10200 IOs, 4237391872 bytes 2024-09-17T23:23:12.245 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10300 IOs, 4323465216 bytes 2024-09-17T23:23:12.292 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ error_code=124 2024-09-17T23:23:12.292 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set -e 2024-09-17T23:23:12.293 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' 124 -eq 124 ']' 2024-09-17T23:23:12.293 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:23:12.293 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:23:12.293 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_40 2024-09-17T23:23:12.293 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_image_and_enable_mirror cluster2 mirror image_40 snapshot 128M 2024-09-17T23:23:12.293 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:23:12.293 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:23:12.293 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:23:12.293 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:23:12.293 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_40 2024-09-17T23:23:12.293 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:23:12.293 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=snapshot 2024-09-17T23:23:12.293 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n snapshot ']' 2024-09-17T23:23:12.293 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:23:12.294 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_image cluster2 mirror image_40 128M 2024-09-17T23:23:12.294 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:23:12.294 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:23:12.294 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:23:12.294 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:23:12.295 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_40 2024-09-17T23:23:12.295 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:23:12.295 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local size=128 2024-09-17T23:23:12.295 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n 128M ']' 2024-09-17T23:23:12.295 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ size=128M 2024-09-17T23:23:12.295 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:23:12.295 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_40 2024-09-17T23:23:12.343 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ is_pool_mirror_mode_image mirror 2024-09-17T23:23:12.343 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:23:12.343 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' image = image ']' 2024-09-17T23:23:12.344 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:23:12.344 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ enable_mirror cluster2 mirror image_40 snapshot 2024-09-17T23:23:12.344 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:23:12.344 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:23:12.344 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_40 2024-09-17T23:23:12.344 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=snapshot 2024-09-17T23:23:12.344 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_40 snapshot 2024-09-17T23:23:12.895 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Mirroring enabled 2024-09-17T23:23:12.905 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 info mirror/image_40 2024-09-17T23:23:12.944 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'image_40': 2024-09-17T23:23:12.944 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 128 MiB in 32 objects 2024-09-17T23:23:12.944 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:23:12.944 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 1 2024-09-17T23:23:12.944 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 1474cfc1e9d7 2024-09-17T23:23:12.944 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.1474cfc1e9d7 2024-09-17T23:23:12.944 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:23:12.944 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-17T23:23:12.944 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:23:12.945 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:23:12.945 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:23:12 2024 2024-09-17T23:23:12.945 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:23:12 2024 2024-09-17T23:23:12.945 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:23:12 2024 2024-09-17T23:23:12.945 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:23:12.945 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:23:12.945 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: 2fcc3df0-03d9-483f-9f1f-787ec2efd74d 2024-09-17T23:23:12.945 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: true 2024-09-17T23:23:12.947 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n '' ']' 2024-09-17T23:23:12.947 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ stress_write_image cluster2 mirror image_40 2024-09-17T23:23:12.947 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:23:12.947 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:23:12.947 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_40 2024-09-17T23:23:12.947 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-17T23:23:12.949 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local duration=13 2024-09-17T23:23:12.949 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set +e 2024-09-17T23:23:12.949 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_40 --debug-rbd=20 --debug-journaler=20 2024-09-17T23:23:13.166 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:100 IOs, 50847744 bytes 2024-09-17T23:23:13.273 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:200 IOs, 73807872 bytes 2024-09-17T23:23:13.410 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:300 IOs, 135420928 bytes 2024-09-17T23:23:13.535 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:400 IOs, 187905024 bytes 2024-09-17T23:23:13.659 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:500 IOs, 232446976 bytes 2024-09-17T23:23:13.782 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:600 IOs, 285991936 bytes 2024-09-17T23:23:13.877 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:700 IOs, 289304576 bytes 2024-09-17T23:23:13.982 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:800 IOs, 314713088 bytes 2024-09-17T23:23:14.095 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:900 IOs, 357411840 bytes 2024-09-17T23:23:14.186 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1000 IOs, 361747456 bytes 2024-09-17T23:23:14.321 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1100 IOs, 434862080 bytes 2024-09-17T23:23:14.449 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1200 IOs, 465704960 bytes 2024-09-17T23:23:14.597 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1300 IOs, 536788992 bytes 2024-09-17T23:23:14.699 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1400 IOs, 556855296 bytes 2024-09-17T23:23:14.838 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1500 IOs, 620527616 bytes 2024-09-17T23:23:14.970 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1600 IOs, 668882944 bytes 2024-09-17T23:23:15.123 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1700 IOs, 748760064 bytes 2024-09-17T23:23:15.279 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1800 IOs, 816215040 bytes 2024-09-17T23:23:15.430 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1900 IOs, 901913600 bytes 2024-09-17T23:23:15.561 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2000 IOs, 941265920 bytes 2024-09-17T23:23:15.682 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2100 IOs, 983877632 bytes 2024-09-17T23:23:15.816 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2200 IOs, 1038394368 bytes 2024-09-17T23:23:15.986 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2300 IOs, 1127550976 bytes 2024-09-17T23:23:16.039 INFO:tasks.ceph.cluster2.osd.0.smithi038.stderr:2024-09-17T23:23:16.036+0000 7fa26a72e640 -1 reset not still connected to 0x55b31c0fdee0 2024-09-17T23:23:16.046 INFO:tasks.ceph.cluster2.osd.0.smithi038.stderr:2024-09-17T23:23:16.039+0000 7fa26a72e640 -1 reset not still connected to 0x55b323997c70 2024-09-17T23:23:16.047 INFO:tasks.ceph.cluster2.osd.0.smithi038.stderr:2024-09-17T23:23:16.043+0000 7fa26a72e640 -1 reset not still connected to 0x55b323e6ac30 2024-09-17T23:23:16.047 INFO:tasks.ceph.cluster2.osd.0.smithi038.stderr:2024-09-17T23:23:16.043+0000 7fa26a72e640 -1 reset not still connected to 0x55b326a65c70 2024-09-17T23:23:16.047 INFO:tasks.ceph.cluster2.osd.0.smithi038.stderr:2024-09-17T23:23:16.043+0000 7fa26a72e640 -1 reset not still connected to 0x55b326c11930 2024-09-17T23:23:16.047 INFO:tasks.ceph.cluster2.osd.0.smithi038.stderr:2024-09-17T23:23:16.043+0000 7fa26a72e640 -1 reset not still connected to 0x55b328baa4e0 2024-09-17T23:23:16.051 INFO:tasks.ceph.cluster2.osd.0.smithi038.stderr:2024-09-17T23:23:16.047+0000 7fa26a72e640 -1 reset not still connected to 0x55b329649c70 2024-09-17T23:23:16.051 INFO:tasks.ceph.cluster2.osd.0.smithi038.stderr:2024-09-17T23:23:16.047+0000 7fa26a72e640 -1 reset not still connected to 0x55b32ddeef70 2024-09-17T23:23:16.051 INFO:tasks.ceph.cluster2.osd.0.smithi038.stderr:2024-09-17T23:23:16.047+0000 7fa26a72e640 -1 reset not still connected to 0x55b32e1005b0 2024-09-17T23:23:16.051 INFO:tasks.ceph.cluster2.osd.0.smithi038.stderr:2024-09-17T23:23:16.047+0000 7fa26a72e640 -1 reset not still connected to 0x55b330241a00 2024-09-17T23:23:16.051 INFO:tasks.ceph.cluster2.osd.0.smithi038.stderr:2024-09-17T23:23:16.047+0000 7fa26a72e640 -1 reset not still connected to 0x55b330e76680 2024-09-17T23:23:16.056 INFO:tasks.ceph.cluster2.osd.0.smithi038.stderr:2024-09-17T23:23:16.053+0000 7fa26a72e640 -1 reset not still connected to 0x55b330e95c70 2024-09-17T23:23:16.155 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2400 IOs, 1205553152 bytes 2024-09-17T23:23:16.463 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2500 IOs, 1260822528 bytes 2024-09-17T23:23:16.621 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2600 IOs, 1319943168 bytes 2024-09-17T23:23:16.729 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2700 IOs, 1355804672 bytes 2024-09-17T23:23:16.851 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2800 IOs, 1391352832 bytes 2024-09-17T23:23:16.956 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2900 IOs, 1414471680 bytes 2024-09-17T23:23:17.129 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3000 IOs, 1511407616 bytes 2024-09-17T23:23:17.322 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3100 IOs, 1621611520 bytes 2024-09-17T23:23:17.446 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3200 IOs, 1667843072 bytes 2024-09-17T23:23:17.602 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3300 IOs, 1739957248 bytes 2024-09-17T23:23:17.789 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3400 IOs, 1850871808 bytes 2024-09-17T23:23:17.908 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3500 IOs, 1911553024 bytes 2024-09-17T23:23:18.060 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3600 IOs, 1964047360 bytes 2024-09-17T23:23:18.217 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3700 IOs, 2050135040 bytes 2024-09-17T23:23:18.332 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3800 IOs, 2075554816 bytes 2024-09-17T23:23:18.414 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3900 IOs, 2076172288 bytes 2024-09-17T23:23:18.522 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4000 IOs, 2107787264 bytes 2024-09-17T23:23:18.659 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4100 IOs, 2158168064 bytes 2024-09-17T23:23:18.787 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4200 IOs, 2204928000 bytes 2024-09-17T23:23:18.908 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4300 IOs, 2247928832 bytes 2024-09-17T23:23:19.095 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4400 IOs, 2353017856 bytes 2024-09-17T23:23:19.240 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4500 IOs, 2440258560 bytes 2024-09-17T23:23:19.396 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4600 IOs, 2496324608 bytes 2024-09-17T23:23:19.556 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4700 IOs, 2589164544 bytes 2024-09-17T23:23:19.745 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4800 IOs, 2694068224 bytes 2024-09-17T23:23:19.857 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4900 IOs, 2727570432 bytes 2024-09-17T23:23:19.948 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5000 IOs, 2736735232 bytes 2024-09-17T23:23:20.061 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5100 IOs, 2772155392 bytes 2024-09-17T23:23:20.178 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5200 IOs, 2800217088 bytes 2024-09-17T23:23:20.277 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5300 IOs, 2816674816 bytes 2024-09-17T23:23:20.372 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5400 IOs, 2829612032 bytes 2024-09-17T23:23:20.486 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5500 IOs, 2863682560 bytes 2024-09-17T23:23:20.602 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5600 IOs, 2906115072 bytes 2024-09-17T23:23:20.694 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5700 IOs, 2913631232 bytes 2024-09-17T23:23:20.821 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5800 IOs, 2964194304 bytes 2024-09-17T23:23:20.972 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5900 IOs, 3033481216 bytes 2024-09-17T23:23:21.057 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6000 IOs, 3034102784 bytes 2024-09-17T23:23:21.171 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6100 IOs, 3067079680 bytes 2024-09-17T23:23:21.366 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6200 IOs, 3196126208 bytes 2024-09-17T23:23:21.524 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6300 IOs, 3258597376 bytes 2024-09-17T23:23:21.648 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6400 IOs, 3304605696 bytes 2024-09-17T23:23:21.737 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6500 IOs, 3313271808 bytes 2024-09-17T23:23:21.827 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6600 IOs, 3322550272 bytes 2024-09-17T23:23:21.968 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6700 IOs, 3382674432 bytes 2024-09-17T23:23:22.068 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6800 IOs, 3405647872 bytes 2024-09-17T23:23:22.179 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6900 IOs, 3436310528 bytes 2024-09-17T23:23:22.306 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7000 IOs, 3484554240 bytes 2024-09-17T23:23:22.430 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7100 IOs, 3528886272 bytes 2024-09-17T23:23:22.536 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7200 IOs, 3557465088 bytes 2024-09-17T23:23:22.625 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7300 IOs, 3561969664 bytes 2024-09-17T23:23:22.726 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7400 IOs, 3585754112 bytes 2024-09-17T23:23:22.888 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7500 IOs, 3667376128 bytes 2024-09-17T23:23:23.031 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7600 IOs, 3730887680 bytes 2024-09-17T23:23:23.136 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7700 IOs, 3752111104 bytes 2024-09-17T23:23:23.265 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7800 IOs, 3805479936 bytes 2024-09-17T23:23:23.390 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7900 IOs, 3843239936 bytes 2024-09-17T23:23:23.493 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8000 IOs, 3877888000 bytes 2024-09-17T23:23:23.599 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8100 IOs, 3887069184 bytes 2024-09-17T23:23:23.705 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8200 IOs, 3917296640 bytes 2024-09-17T23:23:23.894 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8300 IOs, 4017072128 bytes 2024-09-17T23:23:24.014 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8400 IOs, 4055839744 bytes 2024-09-17T23:23:24.147 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8500 IOs, 4108197888 bytes 2024-09-17T23:23:24.279 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8600 IOs, 4164955136 bytes 2024-09-17T23:23:24.386 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8700 IOs, 4190685184 bytes 2024-09-17T23:23:24.504 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8800 IOs, 4230643712 bytes 2024-09-17T23:23:24.619 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8900 IOs, 4264629248 bytes 2024-09-17T23:23:24.741 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9000 IOs, 4324750336 bytes 2024-09-17T23:23:24.886 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9100 IOs, 4365711360 bytes 2024-09-17T23:23:24.992 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9200 IOs, 4389325824 bytes 2024-09-17T23:23:25.098 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9300 IOs, 4410761216 bytes 2024-09-17T23:23:25.186 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9400 IOs, 4418155520 bytes 2024-09-17T23:23:25.349 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9500 IOs, 4495954944 bytes 2024-09-17T23:23:25.510 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9600 IOs, 4575387648 bytes 2024-09-17T23:23:25.633 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9700 IOs, 4620104704 bytes 2024-09-17T23:23:25.740 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9800 IOs, 4638989312 bytes 2024-09-17T23:23:25.890 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9900 IOs, 4709956608 bytes 2024-09-17T23:23:25.955 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ error_code=124 2024-09-17T23:23:25.955 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set -e 2024-09-17T23:23:25.955 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' 124 -eq 124 ']' 2024-09-17T23:23:25.955 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:23:25.955 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:23:25.955 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_41 2024-09-17T23:23:25.955 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_image_and_enable_mirror cluster2 mirror image_41 snapshot 128M 2024-09-17T23:23:25.955 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:23:25.956 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:23:25.956 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:23:25.956 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:23:25.956 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_41 2024-09-17T23:23:25.956 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:23:25.956 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=snapshot 2024-09-17T23:23:25.956 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n snapshot ']' 2024-09-17T23:23:25.956 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:23:25.956 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_image cluster2 mirror image_41 128M 2024-09-17T23:23:25.957 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:23:25.957 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:23:25.957 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:23:25.957 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:23:25.957 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_41 2024-09-17T23:23:25.958 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:23:25.958 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local size=128 2024-09-17T23:23:25.958 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n 128M ']' 2024-09-17T23:23:25.958 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ size=128M 2024-09-17T23:23:25.958 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:23:25.958 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_41 2024-09-17T23:23:26.006 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ is_pool_mirror_mode_image mirror 2024-09-17T23:23:26.007 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:23:26.007 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' image = image ']' 2024-09-17T23:23:26.007 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:23:26.007 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ enable_mirror cluster2 mirror image_41 snapshot 2024-09-17T23:23:26.007 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:23:26.007 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:23:26.007 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_41 2024-09-17T23:23:26.007 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=snapshot 2024-09-17T23:23:26.007 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_41 snapshot 2024-09-17T23:23:26.936 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Mirroring enabled 2024-09-17T23:23:26.951 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 info mirror/image_41 2024-09-17T23:23:26.993 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'image_41': 2024-09-17T23:23:26.993 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 128 MiB in 32 objects 2024-09-17T23:23:26.993 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:23:26.993 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 1 2024-09-17T23:23:26.993 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 14804a37e0f5 2024-09-17T23:23:26.993 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.14804a37e0f5 2024-09-17T23:23:26.993 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:23:26.993 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-17T23:23:26.993 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:23:26.993 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:23:26.994 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:23:25 2024 2024-09-17T23:23:26.994 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:23:25 2024 2024-09-17T23:23:26.994 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:23:25 2024 2024-09-17T23:23:26.994 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:23:26.994 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:23:26.994 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: 7a67405f-a922-4db6-a34d-aef043e8bddf 2024-09-17T23:23:26.994 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: true 2024-09-17T23:23:26.996 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n '' ']' 2024-09-17T23:23:26.997 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ stress_write_image cluster2 mirror image_41 2024-09-17T23:23:26.997 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:23:26.997 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:23:26.997 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_41 2024-09-17T23:23:26.997 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-17T23:23:26.998 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local duration=6 2024-09-17T23:23:26.999 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set +e 2024-09-17T23:23:26.999 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_41 --debug-rbd=20 --debug-journaler=20 2024-09-17T23:23:27.191 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:100 IOs, 23209984 bytes 2024-09-17T23:23:27.340 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:200 IOs, 97402880 bytes 2024-09-17T23:23:27.466 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:300 IOs, 141437952 bytes 2024-09-17T23:23:27.563 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:400 IOs, 152930304 bytes 2024-09-17T23:23:27.686 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:500 IOs, 196066304 bytes 2024-09-17T23:23:27.792 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:600 IOs, 218398720 bytes 2024-09-17T23:23:27.901 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:700 IOs, 243207168 bytes 2024-09-17T23:23:28.028 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:800 IOs, 295248896 bytes 2024-09-17T23:23:28.127 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:900 IOs, 303852544 bytes 2024-09-17T23:23:28.256 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1000 IOs, 354337792 bytes 2024-09-17T23:23:28.353 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1100 IOs, 374920192 bytes 2024-09-17T23:23:28.519 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1200 IOs, 461368320 bytes 2024-09-17T23:23:28.643 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1300 IOs, 495429632 bytes 2024-09-17T23:23:28.805 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1400 IOs, 575211520 bytes 2024-09-17T23:23:28.959 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1500 IOs, 645043200 bytes 2024-09-17T23:23:29.105 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1600 IOs, 708061184 bytes 2024-09-17T23:23:29.208 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1700 IOs, 724296704 bytes 2024-09-17T23:23:29.313 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1800 IOs, 764261376 bytes 2024-09-17T23:23:29.416 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1900 IOs, 764875776 bytes 2024-09-17T23:23:29.547 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2000 IOs, 820854784 bytes 2024-09-17T23:23:29.682 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2100 IOs, 860273664 bytes 2024-09-17T23:23:29.796 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2200 IOs, 894958592 bytes 2024-09-17T23:23:29.928 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2300 IOs, 958793728 bytes 2024-09-17T23:23:30.122 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2400 IOs, 1063602176 bytes 2024-09-17T23:23:30.210 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2500 IOs, 1068995584 bytes 2024-09-17T23:23:30.295 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2600 IOs, 1069613056 bytes 2024-09-17T23:23:30.452 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2700 IOs, 1143981056 bytes 2024-09-17T23:23:30.581 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2800 IOs, 1209138176 bytes 2024-09-17T23:23:30.707 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2900 IOs, 1233302528 bytes 2024-09-17T23:23:30.794 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3000 IOs, 1255417856 bytes 2024-09-17T23:23:30.956 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3100 IOs, 1316583424 bytes 2024-09-17T23:23:31.047 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3200 IOs, 1323999232 bytes 2024-09-17T23:23:31.161 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3300 IOs, 1352622080 bytes 2024-09-17T23:23:31.309 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3400 IOs, 1421154304 bytes 2024-09-17T23:23:31.406 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3500 IOs, 1433369600 bytes 2024-09-17T23:23:31.547 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3600 IOs, 1498532864 bytes 2024-09-17T23:23:31.649 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3700 IOs, 1518164992 bytes 2024-09-17T23:23:31.788 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3800 IOs, 1594561536 bytes 2024-09-17T23:23:31.912 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3900 IOs, 1621022720 bytes 2024-09-17T23:23:32.030 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4000 IOs, 1658458112 bytes 2024-09-17T23:23:32.149 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4100 IOs, 1697304576 bytes 2024-09-17T23:23:32.275 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4200 IOs, 1743419392 bytes 2024-09-17T23:23:32.361 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4300 IOs, 1744028672 bytes 2024-09-17T23:23:32.489 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4400 IOs, 1792243712 bytes 2024-09-17T23:23:32.579 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4500 IOs, 1797114880 bytes 2024-09-17T23:23:32.750 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4600 IOs, 1887022080 bytes 2024-09-17T23:23:32.902 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4700 IOs, 1959494656 bytes 2024-09-17T23:23:32.994 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4800 IOs, 1979137024 bytes 2024-09-17T23:23:33.005 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ error_code=124 2024-09-17T23:23:33.005 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set -e 2024-09-17T23:23:33.005 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' 124 -eq 124 ']' 2024-09-17T23:23:33.005 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:23:33.005 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:23:33.005 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_42 2024-09-17T23:23:33.005 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_image_and_enable_mirror cluster2 mirror image_42 snapshot 128M 2024-09-17T23:23:33.005 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:23:33.005 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:23:33.005 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:23:33.006 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:23:33.006 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_42 2024-09-17T23:23:33.006 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:23:33.006 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=snapshot 2024-09-17T23:23:33.006 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n snapshot ']' 2024-09-17T23:23:33.007 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:23:33.007 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_image cluster2 mirror image_42 128M 2024-09-17T23:23:33.007 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:23:33.007 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:23:33.007 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:23:33.007 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:23:33.007 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_42 2024-09-17T23:23:33.007 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:23:33.007 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local size=128 2024-09-17T23:23:33.007 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n 128M ']' 2024-09-17T23:23:33.008 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ size=128M 2024-09-17T23:23:33.008 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:23:33.008 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_42 2024-09-17T23:23:33.057 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ is_pool_mirror_mode_image mirror 2024-09-17T23:23:33.058 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:23:33.058 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' image = image ']' 2024-09-17T23:23:33.058 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:23:33.058 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ enable_mirror cluster2 mirror image_42 snapshot 2024-09-17T23:23:33.058 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:23:33.058 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:23:33.058 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_42 2024-09-17T23:23:33.058 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=snapshot 2024-09-17T23:23:33.058 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_42 snapshot 2024-09-17T23:23:33.924 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Mirroring enabled 2024-09-17T23:23:33.935 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 info mirror/image_42 2024-09-17T23:23:33.976 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'image_42': 2024-09-17T23:23:33.976 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 128 MiB in 32 objects 2024-09-17T23:23:33.976 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:23:33.976 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 1 2024-09-17T23:23:33.976 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 148c5962b973 2024-09-17T23:23:33.976 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.148c5962b973 2024-09-17T23:23:33.977 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:23:33.977 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-17T23:23:33.977 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:23:33.977 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:23:33.977 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:23:33 2024 2024-09-17T23:23:33.977 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:23:33 2024 2024-09-17T23:23:33.977 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:23:33 2024 2024-09-17T23:23:33.977 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:23:33.977 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:23:33.977 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: 515d2514-6060-40dc-8860-1c0b3ca01bb9 2024-09-17T23:23:33.977 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: true 2024-09-17T23:23:33.980 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n '' ']' 2024-09-17T23:23:33.980 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ stress_write_image cluster2 mirror image_42 2024-09-17T23:23:33.980 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:23:33.980 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:23:33.980 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_42 2024-09-17T23:23:33.980 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-17T23:23:33.982 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local duration=12 2024-09-17T23:23:33.982 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set +e 2024-09-17T23:23:33.982 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_42 --debug-rbd=20 --debug-journaler=20 2024-09-17T23:23:34.216 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:100 IOs, 68508672 bytes 2024-09-17T23:23:34.341 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:200 IOs, 111666176 bytes 2024-09-17T23:23:34.424 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:300 IOs, 112279552 bytes 2024-09-17T23:23:34.512 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:400 IOs, 121065472 bytes 2024-09-17T23:23:34.650 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:500 IOs, 178914304 bytes 2024-09-17T23:23:34.778 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:600 IOs, 224136192 bytes 2024-09-17T23:23:34.867 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:700 IOs, 229202944 bytes 2024-09-17T23:23:35.008 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:800 IOs, 290462720 bytes 2024-09-17T23:23:35.147 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:900 IOs, 351378432 bytes 2024-09-17T23:23:35.243 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1000 IOs, 365353984 bytes 2024-09-17T23:23:35.356 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1100 IOs, 397494272 bytes 2024-09-17T23:23:35.510 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1200 IOs, 469863424 bytes 2024-09-17T23:23:35.643 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1300 IOs, 515745792 bytes 2024-09-17T23:23:35.723 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1400 IOs, 516360192 bytes 2024-09-17T23:23:35.854 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1500 IOs, 566773760 bytes 2024-09-17T23:23:36.000 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1600 IOs, 624215040 bytes 2024-09-17T23:23:36.121 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1700 IOs, 665379840 bytes 2024-09-17T23:23:36.203 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1800 IOs, 665977856 bytes 2024-09-17T23:23:36.363 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1900 IOs, 749540352 bytes 2024-09-17T23:23:36.467 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2000 IOs, 767593472 bytes 2024-09-17T23:23:36.602 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2100 IOs, 820885504 bytes 2024-09-17T23:23:36.722 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2200 IOs, 895172608 bytes 2024-09-17T23:23:36.841 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2300 IOs, 898694144 bytes 2024-09-17T23:23:36.934 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2400 IOs, 909094912 bytes 2024-09-17T23:23:37.016 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2500 IOs, 909731840 bytes 2024-09-17T23:23:37.134 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2600 IOs, 943349760 bytes 2024-09-17T23:23:37.273 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2700 IOs, 1007213568 bytes 2024-09-17T23:23:37.415 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2800 IOs, 1067083776 bytes 2024-09-17T23:23:37.521 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2900 IOs, 1089824768 bytes 2024-09-17T23:23:37.625 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3000 IOs, 1114765312 bytes 2024-09-17T23:23:37.752 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3100 IOs, 1153165312 bytes 2024-09-17T23:23:37.908 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3200 IOs, 1223330816 bytes 2024-09-17T23:23:38.034 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3300 IOs, 1266550784 bytes 2024-09-17T23:23:38.157 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3400 IOs, 1303841792 bytes 2024-09-17T23:23:38.292 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3500 IOs, 1360722944 bytes 2024-09-17T23:23:38.426 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3600 IOs, 1410165760 bytes 2024-09-17T23:23:38.563 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3700 IOs, 1471169536 bytes 2024-09-17T23:23:38.675 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3800 IOs, 1496902656 bytes 2024-09-17T23:23:38.787 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3900 IOs, 1536643072 bytes 2024-09-17T23:23:38.894 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4000 IOs, 1548199936 bytes 2024-09-17T23:23:39.031 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4100 IOs, 1602172928 bytes 2024-09-17T23:23:39.155 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4200 IOs, 1641695232 bytes 2024-09-17T23:23:39.273 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4300 IOs, 1691812864 bytes 2024-09-17T23:23:39.385 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4400 IOs, 1707356160 bytes 2024-09-17T23:23:39.511 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4500 IOs, 1749723136 bytes 2024-09-17T23:23:39.637 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4600 IOs, 1798345728 bytes 2024-09-17T23:23:39.734 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4700 IOs, 1811698688 bytes 2024-09-17T23:23:39.824 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4800 IOs, 1819589632 bytes 2024-09-17T23:23:39.928 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4900 IOs, 1846371328 bytes 2024-09-17T23:23:40.011 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5000 IOs, 1846990848 bytes 2024-09-17T23:23:40.156 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5100 IOs, 1905645568 bytes 2024-09-17T23:23:40.306 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5200 IOs, 1982571520 bytes 2024-09-17T23:23:40.398 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5300 IOs, 1983208448 bytes 2024-09-17T23:23:40.496 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5400 IOs, 2014405632 bytes 2024-09-17T23:23:40.616 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5500 IOs, 2036021248 bytes 2024-09-17T23:23:40.758 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5600 IOs, 2095232000 bytes 2024-09-17T23:23:40.889 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5700 IOs, 2143144960 bytes 2024-09-17T23:23:40.986 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5800 IOs, 2165591040 bytes 2024-09-17T23:23:41.087 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5900 IOs, 2178323456 bytes 2024-09-17T23:23:41.211 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6000 IOs, 2218782720 bytes 2024-09-17T23:23:41.363 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6100 IOs, 2292030464 bytes 2024-09-17T23:23:41.468 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6200 IOs, 2311892992 bytes 2024-09-17T23:23:41.560 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6300 IOs, 2322664448 bytes 2024-09-17T23:23:41.691 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6400 IOs, 2371781632 bytes 2024-09-17T23:23:41.822 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6500 IOs, 2418285568 bytes 2024-09-17T23:23:41.973 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6600 IOs, 2487809024 bytes 2024-09-17T23:23:42.113 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6700 IOs, 2547318784 bytes 2024-09-17T23:23:42.216 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6800 IOs, 2583060480 bytes 2024-09-17T23:23:42.328 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6900 IOs, 2599340032 bytes 2024-09-17T23:23:42.433 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7000 IOs, 2620121088 bytes 2024-09-17T23:23:42.572 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7100 IOs, 2691681280 bytes 2024-09-17T23:23:42.695 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7200 IOs, 2720388096 bytes 2024-09-17T23:23:42.856 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7300 IOs, 2806105088 bytes 2024-09-17T23:23:42.996 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7400 IOs, 2874903552 bytes 2024-09-17T23:23:43.118 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7500 IOs, 2908971008 bytes 2024-09-17T23:23:43.224 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7600 IOs, 2945597440 bytes 2024-09-17T23:23:43.332 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7700 IOs, 2959276032 bytes 2024-09-17T23:23:43.480 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7800 IOs, 3023969280 bytes 2024-09-17T23:23:43.638 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7900 IOs, 3102774272 bytes 2024-09-17T23:23:43.809 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8000 IOs, 3196107776 bytes 2024-09-17T23:23:43.936 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8100 IOs, 3241179136 bytes 2024-09-17T23:23:44.017 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8200 IOs, 3241802752 bytes 2024-09-17T23:23:44.145 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8300 IOs, 3289665536 bytes 2024-09-17T23:23:44.258 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8400 IOs, 3324755968 bytes 2024-09-17T23:23:44.380 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8500 IOs, 3368595456 bytes 2024-09-17T23:23:44.499 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8600 IOs, 3404066816 bytes 2024-09-17T23:23:44.673 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8700 IOs, 3500457984 bytes 2024-09-17T23:23:44.794 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8800 IOs, 3537431552 bytes 2024-09-17T23:23:44.933 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8900 IOs, 3614361600 bytes 2024-09-17T23:23:45.080 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9000 IOs, 3664539648 bytes 2024-09-17T23:23:45.191 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9100 IOs, 3705202688 bytes 2024-09-17T23:23:45.309 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9200 IOs, 3732966400 bytes 2024-09-17T23:23:45.446 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9300 IOs, 3792623616 bytes 2024-09-17T23:23:45.548 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9400 IOs, 3810081792 bytes 2024-09-17T23:23:45.692 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9500 IOs, 3873174528 bytes 2024-09-17T23:23:45.849 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9600 IOs, 3944237056 bytes 2024-09-17T23:23:45.974 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9700 IOs, 3992454144 bytes 2024-09-17T23:23:45.989 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ error_code=124 2024-09-17T23:23:45.989 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set -e 2024-09-17T23:23:45.990 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' 124 -eq 124 ']' 2024-09-17T23:23:45.990 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:23:45.990 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:23:45.990 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_43 2024-09-17T23:23:45.990 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_image_and_enable_mirror cluster2 mirror image_43 snapshot 128M 2024-09-17T23:23:45.990 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:23:45.990 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:23:45.990 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:23:45.990 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:23:45.990 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_43 2024-09-17T23:23:45.990 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:23:45.990 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=snapshot 2024-09-17T23:23:45.990 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n snapshot ']' 2024-09-17T23:23:45.991 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:23:45.991 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_image cluster2 mirror image_43 128M 2024-09-17T23:23:45.991 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:23:45.991 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:23:45.991 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:23:45.991 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:23:45.991 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_43 2024-09-17T23:23:45.991 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:23:45.991 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local size=128 2024-09-17T23:23:45.991 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n 128M ']' 2024-09-17T23:23:45.991 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ size=128M 2024-09-17T23:23:45.991 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:23:45.991 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_43 2024-09-17T23:23:46.042 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ is_pool_mirror_mode_image mirror 2024-09-17T23:23:46.042 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:23:46.042 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' image = image ']' 2024-09-17T23:23:46.042 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:23:46.042 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ enable_mirror cluster2 mirror image_43 snapshot 2024-09-17T23:23:46.042 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:23:46.042 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:23:46.042 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_43 2024-09-17T23:23:46.043 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=snapshot 2024-09-17T23:23:46.043 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_43 snapshot 2024-09-17T23:23:46.957 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Mirroring enabled 2024-09-17T23:23:46.968 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 info mirror/image_43 2024-09-17T23:23:47.009 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'image_43': 2024-09-17T23:23:47.009 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 128 MiB in 32 objects 2024-09-17T23:23:47.009 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:23:47.009 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 1 2024-09-17T23:23:47.009 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 149877b08816 2024-09-17T23:23:47.009 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.149877b08816 2024-09-17T23:23:47.009 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:23:47.010 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-17T23:23:47.010 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:23:47.010 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:23:47.010 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:23:46 2024 2024-09-17T23:23:47.010 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:23:46 2024 2024-09-17T23:23:47.010 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:23:46 2024 2024-09-17T23:23:47.010 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:23:47.010 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:23:47.010 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: 65d764ca-56da-405c-b68b-d13fb878db55 2024-09-17T23:23:47.010 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: true 2024-09-17T23:23:47.012 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n '' ']' 2024-09-17T23:23:47.013 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ stress_write_image cluster2 mirror image_43 2024-09-17T23:23:47.013 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:23:47.013 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:23:47.013 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_43 2024-09-17T23:23:47.013 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-17T23:23:47.015 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local duration=9 2024-09-17T23:23:47.015 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set +e 2024-09-17T23:23:47.015 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_43 --debug-rbd=20 --debug-journaler=20 2024-09-17T23:23:47.216 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:100 IOs, 33386496 bytes 2024-09-17T23:23:47.351 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:200 IOs, 82233344 bytes 2024-09-17T23:23:47.448 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:300 IOs, 102999040 bytes 2024-09-17T23:23:47.597 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:400 IOs, 177230848 bytes 2024-09-17T23:23:47.722 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:500 IOs, 210157568 bytes 2024-09-17T23:23:47.805 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:600 IOs, 210742272 bytes 2024-09-17T23:23:47.972 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:700 IOs, 301334528 bytes 2024-09-17T23:23:48.116 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:800 IOs, 378284032 bytes 2024-09-17T23:23:48.260 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:900 IOs, 418881536 bytes 2024-09-17T23:23:48.373 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1000 IOs, 469872640 bytes 2024-09-17T23:23:48.510 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1100 IOs, 506662912 bytes 2024-09-17T23:23:48.616 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1200 IOs, 530936832 bytes 2024-09-17T23:23:48.756 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1300 IOs, 591912960 bytes 2024-09-17T23:23:48.925 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1400 IOs, 678278144 bytes 2024-09-17T23:23:49.051 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1500 IOs, 721554432 bytes 2024-09-17T23:23:49.204 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1600 IOs, 788247552 bytes 2024-09-17T23:23:49.288 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1700 IOs, 788864000 bytes 2024-09-17T23:23:49.411 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1800 IOs, 844880896 bytes 2024-09-17T23:23:49.550 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1900 IOs, 889606144 bytes 2024-09-17T23:23:49.697 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2000 IOs, 954249216 bytes 2024-09-17T23:23:49.831 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2100 IOs, 1003840512 bytes 2024-09-17T23:23:49.911 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2200 IOs, 1004920832 bytes 2024-09-17T23:23:49.936 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-09-17T23:23:49.937 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-09-17T23:23:49.937 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2024-09-17T23:23:49.938 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi038.stderr:2024-09-17T23:23:49.935+0000 7f914fea2640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 45927) UID: 1000 2024-09-17T23:23:50.026 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2300 IOs, 1021774848 bytes 2024-09-17T23:23:50.129 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2400 IOs, 1038202880 bytes 2024-09-17T23:23:50.231 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2500 IOs, 1056319488 bytes 2024-09-17T23:23:50.345 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2600 IOs, 1090234368 bytes 2024-09-17T23:23:50.494 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2700 IOs, 1157133312 bytes 2024-09-17T23:23:50.585 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2800 IOs, 1170560000 bytes 2024-09-17T23:23:50.669 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2900 IOs, 1171169280 bytes 2024-09-17T23:23:50.772 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3000 IOs, 1195542528 bytes 2024-09-17T23:23:50.896 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3100 IOs, 1233206272 bytes 2024-09-17T23:23:51.008 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3200 IOs, 1265343488 bytes 2024-09-17T23:23:51.130 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3300 IOs, 1331051520 bytes 2024-09-17T23:23:51.317 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3400 IOs, 1414079488 bytes 2024-09-17T23:23:51.423 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3500 IOs, 1440117760 bytes 2024-09-17T23:23:51.620 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3600 IOs, 1555590144 bytes 2024-09-17T23:23:51.743 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3700 IOs, 1593713664 bytes 2024-09-17T23:23:51.933 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3800 IOs, 1701901312 bytes 2024-09-17T23:23:52.065 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3900 IOs, 1769317376 bytes 2024-09-17T23:23:52.176 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4000 IOs, 1779427328 bytes 2024-09-17T23:23:52.268 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4100 IOs, 1795202048 bytes 2024-09-17T23:23:52.350 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4200 IOs, 1795818496 bytes 2024-09-17T23:23:52.512 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4300 IOs, 1881838592 bytes 2024-09-17T23:23:52.664 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4400 IOs, 1953008640 bytes 2024-09-17T23:23:52.771 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4500 IOs, 1978428416 bytes 2024-09-17T23:23:52.871 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4600 IOs, 1991800832 bytes 2024-09-17T23:23:52.966 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4700 IOs, 2004813824 bytes 2024-09-17T23:23:53.081 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4800 IOs, 2037954560 bytes 2024-09-17T23:23:53.217 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4900 IOs, 2096634880 bytes 2024-09-17T23:23:53.359 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5000 IOs, 2163006464 bytes 2024-09-17T23:23:53.445 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5100 IOs, 2166854656 bytes 2024-09-17T23:23:53.560 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5200 IOs, 2203142144 bytes 2024-09-17T23:23:53.705 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5300 IOs, 2257121280 bytes 2024-09-17T23:23:53.821 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5400 IOs, 2292130816 bytes 2024-09-17T23:23:53.953 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5500 IOs, 2337371136 bytes 2024-09-17T23:23:54.071 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5600 IOs, 2373909504 bytes 2024-09-17T23:23:54.245 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5700 IOs, 2470125568 bytes 2024-09-17T23:23:54.329 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5800 IOs, 2474085376 bytes 2024-09-17T23:23:54.487 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5900 IOs, 2554468352 bytes 2024-09-17T23:23:54.808 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6000 IOs, 2625659904 bytes 2024-09-17T23:23:54.984 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6100 IOs, 2719481856 bytes 2024-09-17T23:23:55.140 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6200 IOs, 2796049408 bytes 2024-09-17T23:23:55.239 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6300 IOs, 2814112768 bytes 2024-09-17T23:23:55.380 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6400 IOs, 2875332608 bytes 2024-09-17T23:23:55.509 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6500 IOs, 2932336640 bytes 2024-09-17T23:23:55.617 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6600 IOs, 2946827264 bytes 2024-09-17T23:23:55.755 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6700 IOs, 2995486720 bytes 2024-09-17T23:23:55.875 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6800 IOs, 3030429696 bytes 2024-09-17T23:23:55.982 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6900 IOs, 3051337728 bytes 2024-09-17T23:23:56.020 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ error_code=124 2024-09-17T23:23:56.021 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set -e 2024-09-17T23:23:56.021 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' 124 -eq 124 ']' 2024-09-17T23:23:56.021 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:23:56.021 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:23:56.021 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_44 2024-09-17T23:23:56.021 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_image_and_enable_mirror cluster2 mirror image_44 snapshot 128M 2024-09-17T23:23:56.021 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:23:56.021 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:23:56.021 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:23:56.021 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:23:56.021 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_44 2024-09-17T23:23:56.021 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:23:56.022 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=snapshot 2024-09-17T23:23:56.022 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n snapshot ']' 2024-09-17T23:23:56.022 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:23:56.022 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_image cluster2 mirror image_44 128M 2024-09-17T23:23:56.022 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:23:56.022 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:23:56.022 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:23:56.022 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:23:56.022 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_44 2024-09-17T23:23:56.022 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:23:56.022 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local size=128 2024-09-17T23:23:56.022 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n 128M ']' 2024-09-17T23:23:56.022 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ size=128M 2024-09-17T23:23:56.022 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:23:56.023 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_44 2024-09-17T23:23:56.073 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ is_pool_mirror_mode_image mirror 2024-09-17T23:23:56.074 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:23:56.074 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' image = image ']' 2024-09-17T23:23:56.074 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:23:56.074 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ enable_mirror cluster2 mirror image_44 snapshot 2024-09-17T23:23:56.074 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:23:56.074 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:23:56.074 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_44 2024-09-17T23:23:56.074 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=snapshot 2024-09-17T23:23:56.074 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_44 snapshot 2024-09-17T23:23:56.938 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-09-17T23:23:56.938 INFO:teuthology.orchestra.run:waiting for 600 2024-09-17T23:23:56.938 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-09-17T23:23:56.938 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-09-17T23:23:56.939 DEBUG:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-09-17T23:23:56.944 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-09-17T23:23:56.944 INFO:tasks.rbd_mirror_thrash:waiting for 68 secs before thrashing 2024-09-17T23:23:56.977 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Mirroring enabled 2024-09-17T23:23:56.990 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 info mirror/image_44 2024-09-17T23:23:57.031 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'image_44': 2024-09-17T23:23:57.031 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 128 MiB in 32 objects 2024-09-17T23:23:57.031 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:23:57.031 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 1 2024-09-17T23:23:57.031 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 14a4c2ad2199 2024-09-17T23:23:57.032 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.14a4c2ad2199 2024-09-17T23:23:57.032 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:23:57.032 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-17T23:23:57.032 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:23:57.032 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:23:57.032 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:23:56 2024 2024-09-17T23:23:57.032 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:23:56 2024 2024-09-17T23:23:57.032 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:23:56 2024 2024-09-17T23:23:57.032 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:23:57.033 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:23:57.033 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: 1c859b2d-e379-40d7-bc46-427fb4a14245 2024-09-17T23:23:57.033 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: true 2024-09-17T23:23:57.035 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n '' ']' 2024-09-17T23:23:57.036 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ stress_write_image cluster2 mirror image_44 2024-09-17T23:23:57.036 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:23:57.036 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:23:57.036 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_44 2024-09-17T23:23:57.036 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-17T23:23:57.037 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local duration=6 2024-09-17T23:23:57.038 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set +e 2024-09-17T23:23:57.038 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_44 --debug-rbd=20 --debug-journaler=20 2024-09-17T23:23:57.214 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:100 IOs, 5790720 bytes 2024-09-17T23:23:57.338 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:200 IOs, 41914368 bytes 2024-09-17T23:23:57.508 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:300 IOs, 109379584 bytes 2024-09-17T23:23:57.613 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:400 IOs, 127922176 bytes 2024-09-17T23:23:57.743 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:500 IOs, 173011968 bytes 2024-09-17T23:23:57.855 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:600 IOs, 204594176 bytes 2024-09-17T23:23:58.022 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:700 IOs, 289992704 bytes 2024-09-17T23:23:58.111 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:800 IOs, 313910272 bytes 2024-09-17T23:23:58.232 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:900 IOs, 333692928 bytes 2024-09-17T23:23:58.346 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1000 IOs, 372728832 bytes 2024-09-17T23:23:58.546 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1100 IOs, 487723008 bytes 2024-09-17T23:23:58.679 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1200 IOs, 539427840 bytes 2024-09-17T23:23:58.794 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1300 IOs, 569993216 bytes 2024-09-17T23:23:58.881 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1400 IOs, 575203328 bytes 2024-09-17T23:23:59.025 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1500 IOs, 633882624 bytes 2024-09-17T23:23:59.167 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1600 IOs, 686255104 bytes 2024-09-17T23:23:59.268 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1700 IOs, 710158336 bytes 2024-09-17T23:23:59.393 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1800 IOs, 767491072 bytes 2024-09-17T23:23:59.574 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1900 IOs, 850795520 bytes 2024-09-17T23:23:59.697 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2000 IOs, 889085952 bytes 2024-09-17T23:23:59.834 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2100 IOs, 940529664 bytes 2024-09-17T23:23:59.960 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2200 IOs, 981989376 bytes 2024-09-17T23:24:00.085 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2300 IOs, 1022674944 bytes 2024-09-17T23:24:00.205 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2400 IOs, 1057079296 bytes 2024-09-17T23:24:00.370 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2500 IOs, 1137744896 bytes 2024-09-17T23:24:00.524 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2600 IOs, 1208561664 bytes 2024-09-17T23:24:00.652 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2700 IOs, 1260797952 bytes 2024-09-17T23:24:00.788 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2800 IOs, 1305441280 bytes 2024-09-17T23:24:00.984 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2900 IOs, 1409312768 bytes 2024-09-17T23:24:01.132 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3000 IOs, 1478080512 bytes 2024-09-17T23:24:01.278 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3100 IOs, 1539113984 bytes 2024-09-17T23:24:01.398 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3200 IOs, 1577954304 bytes 2024-09-17T23:24:01.561 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3300 IOs, 1612512256 bytes 2024-09-17T23:24:01.662 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3400 IOs, 1647891456 bytes 2024-09-17T23:24:01.823 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3500 IOs, 1711870976 bytes 2024-09-17T23:24:01.907 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3600 IOs, 1712495616 bytes 2024-09-17T23:24:02.083 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3700 IOs, 1802906624 bytes 2024-09-17T23:24:02.254 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3800 IOs, 1850097664 bytes 2024-09-17T23:24:02.361 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3900 IOs, 1873353728 bytes 2024-09-17T23:24:02.447 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4000 IOs, 1876775936 bytes 2024-09-17T23:24:02.543 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4100 IOs, 1907760128 bytes 2024-09-17T23:24:02.681 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4200 IOs, 1946042368 bytes 2024-09-17T23:24:02.800 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4300 IOs, 1980277760 bytes 2024-09-17T23:24:02.986 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4400 IOs, 2080259072 bytes 2024-09-17T23:24:03.043 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ error_code=124 2024-09-17T23:24:03.043 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set -e 2024-09-17T23:24:03.043 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' 124 -eq 124 ']' 2024-09-17T23:24:03.043 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:24:03.043 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:24:03.043 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_45 2024-09-17T23:24:03.044 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_image_and_enable_mirror cluster2 mirror image_45 snapshot 128M 2024-09-17T23:24:03.044 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:24:03.044 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:24:03.044 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:24:03.044 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:24:03.044 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_45 2024-09-17T23:24:03.044 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:24:03.044 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=snapshot 2024-09-17T23:24:03.044 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n snapshot ']' 2024-09-17T23:24:03.044 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:24:03.044 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_image cluster2 mirror image_45 128M 2024-09-17T23:24:03.044 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:24:03.044 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:24:03.045 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:24:03.045 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:24:03.045 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_45 2024-09-17T23:24:03.045 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:24:03.045 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local size=128 2024-09-17T23:24:03.045 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n 128M ']' 2024-09-17T23:24:03.045 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ size=128M 2024-09-17T23:24:03.045 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:24:03.045 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_45 2024-09-17T23:24:03.095 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ is_pool_mirror_mode_image mirror 2024-09-17T23:24:03.095 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:24:03.096 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' image = image ']' 2024-09-17T23:24:03.096 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:24:03.096 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ enable_mirror cluster2 mirror image_45 snapshot 2024-09-17T23:24:03.096 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:24:03.096 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:24:03.096 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_45 2024-09-17T23:24:03.096 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=snapshot 2024-09-17T23:24:03.096 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_45 snapshot 2024-09-17T23:24:03.994 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Mirroring enabled 2024-09-17T23:24:04.006 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 info mirror/image_45 2024-09-17T23:24:04.248 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'image_45': 2024-09-17T23:24:04.248 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 128 MiB in 32 objects 2024-09-17T23:24:04.248 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:24:04.248 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 1 2024-09-17T23:24:04.248 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 14b9c7befc84 2024-09-17T23:24:04.248 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.14b9c7befc84 2024-09-17T23:24:04.248 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:24:04.248 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-17T23:24:04.248 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:24:04.248 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:24:04.248 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:24:03 2024 2024-09-17T23:24:04.248 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:24:03 2024 2024-09-17T23:24:04.249 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:24:03 2024 2024-09-17T23:24:04.249 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:24:04.249 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:24:04.249 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: a44d8381-af32-4227-90fa-0531c242a7f7 2024-09-17T23:24:04.249 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: true 2024-09-17T23:24:04.252 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n '' ']' 2024-09-17T23:24:04.252 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ stress_write_image cluster2 mirror image_45 2024-09-17T23:24:04.252 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:24:04.252 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:24:04.252 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_45 2024-09-17T23:24:04.252 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-17T23:24:04.254 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local duration=7 2024-09-17T23:24:04.254 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set +e 2024-09-17T23:24:04.254 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_45 --debug-rbd=20 --debug-journaler=20 2024-09-17T23:24:04.447 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:100 IOs, 17583104 bytes 2024-09-17T23:24:04.577 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:200 IOs, 67574784 bytes 2024-09-17T23:24:04.711 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:300 IOs, 148917248 bytes 2024-09-17T23:24:04.859 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:400 IOs, 192896000 bytes 2024-09-17T23:24:04.990 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:500 IOs, 241933312 bytes 2024-09-17T23:24:05.118 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:600 IOs, 284358656 bytes 2024-09-17T23:24:05.242 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:700 IOs, 342194176 bytes 2024-09-17T23:24:05.380 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:800 IOs, 389601280 bytes 2024-09-17T23:24:05.476 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:900 IOs, 407949312 bytes 2024-09-17T23:24:05.561 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1000 IOs, 414185472 bytes 2024-09-17T23:24:05.660 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1100 IOs, 436469760 bytes 2024-09-17T23:24:05.806 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1200 IOs, 493753344 bytes 2024-09-17T23:24:05.941 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1300 IOs, 550349824 bytes 2024-09-17T23:24:06.060 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1400 IOs, 579060736 bytes 2024-09-17T23:24:06.221 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1500 IOs, 668991488 bytes 2024-09-17T23:24:06.326 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1600 IOs, 694041600 bytes 2024-09-17T23:24:06.413 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1700 IOs, 717216768 bytes 2024-09-17T23:24:06.557 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1800 IOs, 758467584 bytes 2024-09-17T23:24:06.649 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1900 IOs, 772287488 bytes 2024-09-17T23:24:06.821 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2000 IOs, 867046400 bytes 2024-09-17T23:24:06.941 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2100 IOs, 910867456 bytes 2024-09-17T23:24:07.060 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2200 IOs, 953887744 bytes 2024-09-17T23:24:07.214 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2300 IOs, 1023800320 bytes 2024-09-17T23:24:07.334 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2400 IOs, 1066081280 bytes 2024-09-17T23:24:07.480 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2500 IOs, 1139312640 bytes 2024-09-17T23:24:07.571 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2600 IOs, 1161560064 bytes 2024-09-17T23:24:07.675 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2700 IOs, 1173842944 bytes 2024-09-17T23:24:07.776 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2800 IOs, 1193904128 bytes 2024-09-17T23:24:07.876 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2900 IOs, 1223502848 bytes 2024-09-17T23:24:08.068 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3000 IOs, 1323890688 bytes 2024-09-17T23:24:08.164 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3100 IOs, 1342612480 bytes 2024-09-17T23:24:08.275 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3200 IOs, 1370921984 bytes 2024-09-17T23:24:08.433 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3300 IOs, 1451504640 bytes 2024-09-17T23:24:08.550 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3400 IOs, 1492631552 bytes 2024-09-17T23:24:08.634 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3500 IOs, 1494509568 bytes 2024-09-17T23:24:08.714 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3600 IOs, 1500112896 bytes 2024-09-17T23:24:08.852 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3700 IOs, 1568376832 bytes 2024-09-17T23:24:09.020 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3800 IOs, 1637379072 bytes 2024-09-17T23:24:09.158 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3900 IOs, 1693472768 bytes 2024-09-17T23:24:09.272 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4000 IOs, 1736863744 bytes 2024-09-17T23:24:09.429 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4100 IOs, 1809294336 bytes 2024-09-17T23:24:09.557 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4200 IOs, 1858132992 bytes 2024-09-17T23:24:09.690 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4300 IOs, 1916158976 bytes 2024-09-17T23:24:09.833 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4400 IOs, 1977308160 bytes 2024-09-17T23:24:09.968 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4500 IOs, 2037645312 bytes 2024-09-17T23:24:10.120 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4600 IOs, 2114123776 bytes 2024-09-17T23:24:10.232 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4700 IOs, 2142219264 bytes 2024-09-17T23:24:10.336 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4800 IOs, 2165327872 bytes 2024-09-17T23:24:10.429 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4900 IOs, 2177386496 bytes 2024-09-17T23:24:10.553 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5000 IOs, 2222517248 bytes 2024-09-17T23:24:10.650 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5100 IOs, 2238610432 bytes 2024-09-17T23:24:10.737 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5200 IOs, 2246804480 bytes 2024-09-17T23:24:10.838 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5300 IOs, 2266603520 bytes 2024-09-17T23:24:10.988 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5400 IOs, 2338282496 bytes 2024-09-17T23:24:11.124 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5500 IOs, 2397584384 bytes 2024-09-17T23:24:11.249 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5600 IOs, 2436734976 bytes 2024-09-17T23:24:11.260 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ error_code=124 2024-09-17T23:24:11.260 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set -e 2024-09-17T23:24:11.260 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' 124 -eq 124 ']' 2024-09-17T23:24:11.260 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:24:11.260 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:24:11.260 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_46 2024-09-17T23:24:11.261 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_image_and_enable_mirror cluster2 mirror image_46 snapshot 128M 2024-09-17T23:24:11.261 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:24:11.261 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:24:11.261 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:24:11.261 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:24:11.261 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_46 2024-09-17T23:24:11.261 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:24:11.261 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=snapshot 2024-09-17T23:24:11.261 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n snapshot ']' 2024-09-17T23:24:11.261 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:24:11.261 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_image cluster2 mirror image_46 128M 2024-09-17T23:24:11.262 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:24:11.262 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:24:11.262 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:24:11.262 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:24:11.262 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_46 2024-09-17T23:24:11.262 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:24:11.262 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local size=128 2024-09-17T23:24:11.262 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n 128M ']' 2024-09-17T23:24:11.262 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ size=128M 2024-09-17T23:24:11.262 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:24:11.262 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_46 2024-09-17T23:24:11.311 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ is_pool_mirror_mode_image mirror 2024-09-17T23:24:11.311 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:24:11.311 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' image = image ']' 2024-09-17T23:24:11.311 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:24:11.311 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ enable_mirror cluster2 mirror image_46 snapshot 2024-09-17T23:24:11.311 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:24:11.311 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:24:11.311 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_46 2024-09-17T23:24:11.311 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=snapshot 2024-09-17T23:24:11.312 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_46 snapshot 2024-09-17T23:24:12.006 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Mirroring enabled 2024-09-17T23:24:12.018 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 info mirror/image_46 2024-09-17T23:24:12.057 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'image_46': 2024-09-17T23:24:12.057 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 128 MiB in 32 objects 2024-09-17T23:24:12.057 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:24:12.058 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 1 2024-09-17T23:24:12.058 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 14c543f7b1d7 2024-09-17T23:24:12.058 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.14c543f7b1d7 2024-09-17T23:24:12.058 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:24:12.058 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-17T23:24:12.058 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:24:12.058 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:24:12.058 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:24:11 2024 2024-09-17T23:24:12.058 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:24:11 2024 2024-09-17T23:24:12.058 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:24:11 2024 2024-09-17T23:24:12.059 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:24:12.059 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:24:12.059 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: bb8f076b-e4d3-47ce-9e64-72dd50f71318 2024-09-17T23:24:12.059 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: true 2024-09-17T23:24:12.061 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n '' ']' 2024-09-17T23:24:12.061 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ stress_write_image cluster2 mirror image_46 2024-09-17T23:24:12.061 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:24:12.061 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:24:12.061 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_46 2024-09-17T23:24:12.061 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-17T23:24:12.063 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local duration=6 2024-09-17T23:24:12.063 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set +e 2024-09-17T23:24:12.063 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_46 --debug-rbd=20 --debug-journaler=20 2024-09-17T23:24:12.325 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:100 IOs, 97704960 bytes 2024-09-17T23:24:12.458 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:200 IOs, 144143360 bytes 2024-09-17T23:24:12.563 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:300 IOs, 192358400 bytes 2024-09-17T23:24:12.688 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:400 IOs, 220780544 bytes 2024-09-17T23:24:12.838 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:500 IOs, 290455552 bytes 2024-09-17T23:24:12.943 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:600 IOs, 315339776 bytes 2024-09-17T23:24:13.049 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:700 IOs, 337880064 bytes 2024-09-17T23:24:13.161 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:800 IOs, 373075968 bytes 2024-09-17T23:24:13.263 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:900 IOs, 390055936 bytes 2024-09-17T23:24:13.373 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1000 IOs, 418806784 bytes 2024-09-17T23:24:13.490 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1100 IOs, 454900736 bytes 2024-09-17T23:24:13.600 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1200 IOs, 477201408 bytes 2024-09-17T23:24:13.741 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1300 IOs, 538244096 bytes 2024-09-17T23:24:13.823 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1400 IOs, 539310080 bytes 2024-09-17T23:24:13.940 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1500 IOs, 576774144 bytes 2024-09-17T23:24:14.113 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1600 IOs, 659903488 bytes 2024-09-17T23:24:14.273 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1700 IOs, 737781760 bytes 2024-09-17T23:24:14.376 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1800 IOs, 761802752 bytes 2024-09-17T23:24:14.459 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1900 IOs, 763247616 bytes 2024-09-17T23:24:14.563 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2000 IOs, 789713920 bytes 2024-09-17T23:24:14.660 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2100 IOs, 803038208 bytes 2024-09-17T23:24:14.784 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2200 IOs, 850625536 bytes 2024-09-17T23:24:14.869 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2300 IOs, 851234816 bytes 2024-09-17T23:24:14.953 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2400 IOs, 854181888 bytes 2024-09-17T23:24:15.081 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2500 IOs, 899481600 bytes 2024-09-17T23:24:15.179 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2600 IOs, 913000448 bytes 2024-09-17T23:24:15.313 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2700 IOs, 978828288 bytes 2024-09-17T23:24:15.465 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2800 IOs, 1032650752 bytes 2024-09-17T23:24:15.638 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2900 IOs, 1151055872 bytes 2024-09-17T23:24:15.764 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3000 IOs, 1179199488 bytes 2024-09-17T23:24:15.844 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3100 IOs, 1194039296 bytes 2024-09-17T23:24:16.005 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3200 IOs, 1256888320 bytes 2024-09-17T23:24:16.101 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3300 IOs, 1269447680 bytes 2024-09-17T23:24:16.237 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3400 IOs, 1323476992 bytes 2024-09-17T23:24:16.370 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3500 IOs, 1378139136 bytes 2024-09-17T23:24:16.477 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3600 IOs, 1402854400 bytes 2024-09-17T23:24:16.608 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3700 IOs, 1473033216 bytes 2024-09-17T23:24:16.729 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3800 IOs, 1494342656 bytes 2024-09-17T23:24:16.878 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3900 IOs, 1556323328 bytes 2024-09-17T23:24:16.996 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4000 IOs, 1589652480 bytes 2024-09-17T23:24:17.081 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4100 IOs, 1590269952 bytes 2024-09-17T23:24:17.210 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4200 IOs, 1631052800 bytes 2024-09-17T23:24:17.334 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4300 IOs, 1674419200 bytes 2024-09-17T23:24:17.475 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4400 IOs, 1735486464 bytes 2024-09-17T23:24:17.580 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4500 IOs, 1768102912 bytes 2024-09-17T23:24:17.697 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4600 IOs, 1789802496 bytes 2024-09-17T23:24:17.804 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4700 IOs, 1818796032 bytes 2024-09-17T23:24:17.837 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.8951098448793641) > weight (0.25) 2024-09-17T23:24:17.837 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.7446983171880014) > weight (0.25) 2024-09-17T23:24:17.837 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2024-09-17T23:24:17.838 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2024-09-17T23:24:17.838 INFO:tasks.rbd_mirror_thrash:waiting for 9 secs before reviving daemons 2024-09-17T23:24:17.838 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi038.stderr:2024-09-17T23:24:17.836+0000 7f5421f29640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 36657) UID: 1000 2024-09-17T23:24:17.924 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4800 IOs, 1847144448 bytes 2024-09-17T23:24:18.070 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ error_code=124 2024-09-17T23:24:18.070 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set -e 2024-09-17T23:24:18.070 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' 124 -eq 124 ']' 2024-09-17T23:24:18.070 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:24:18.070 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:24:18.070 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_47 2024-09-17T23:24:18.070 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_image_and_enable_mirror cluster2 mirror image_47 snapshot 128M 2024-09-17T23:24:18.070 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:24:18.071 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:24:18.071 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:24:18.071 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:24:18.071 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_47 2024-09-17T23:24:18.071 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:24:18.071 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=snapshot 2024-09-17T23:24:18.071 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n snapshot ']' 2024-09-17T23:24:18.071 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:24:18.071 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_image cluster2 mirror image_47 128M 2024-09-17T23:24:18.071 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:24:18.071 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:24:18.071 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:24:18.071 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:24:18.071 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_47 2024-09-17T23:24:18.072 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:24:18.072 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local size=128 2024-09-17T23:24:18.072 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n 128M ']' 2024-09-17T23:24:18.072 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ size=128M 2024-09-17T23:24:18.072 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:24:18.072 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_47 2024-09-17T23:24:18.121 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ is_pool_mirror_mode_image mirror 2024-09-17T23:24:18.121 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:24:18.121 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' image = image ']' 2024-09-17T23:24:18.121 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:24:18.122 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ enable_mirror cluster2 mirror image_47 snapshot 2024-09-17T23:24:18.122 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:24:18.122 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:24:18.122 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_47 2024-09-17T23:24:18.122 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=snapshot 2024-09-17T23:24:18.122 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_47 snapshot 2024-09-17T23:24:19.040 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Mirroring enabled 2024-09-17T23:24:19.052 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 info mirror/image_47 2024-09-17T23:24:19.093 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'image_47': 2024-09-17T23:24:19.093 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 128 MiB in 32 objects 2024-09-17T23:24:19.093 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:24:19.093 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 1 2024-09-17T23:24:19.093 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 14d042a4b49a 2024-09-17T23:24:19.093 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.14d042a4b49a 2024-09-17T23:24:19.093 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:24:19.093 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-17T23:24:19.093 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:24:19.093 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:24:19.093 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:24:18 2024 2024-09-17T23:24:19.094 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:24:18 2024 2024-09-17T23:24:19.094 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:24:18 2024 2024-09-17T23:24:19.094 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:24:19.094 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:24:19.094 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: 04e3e2c8-dba3-4aac-a2db-a1e1fd0c90d3 2024-09-17T23:24:19.094 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: true 2024-09-17T23:24:19.096 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n '' ']' 2024-09-17T23:24:19.097 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ stress_write_image cluster2 mirror image_47 2024-09-17T23:24:19.097 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:24:19.097 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:24:19.097 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_47 2024-09-17T23:24:19.097 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-17T23:24:19.098 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local duration=5 2024-09-17T23:24:19.099 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set +e 2024-09-17T23:24:19.099 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_47 --debug-rbd=20 --debug-journaler=20 2024-09-17T23:24:19.333 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:100 IOs, 70519808 bytes 2024-09-17T23:24:19.438 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:200 IOs, 98792448 bytes 2024-09-17T23:24:19.541 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:300 IOs, 126813184 bytes 2024-09-17T23:24:19.656 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:400 IOs, 160768000 bytes 2024-09-17T23:24:19.816 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:500 IOs, 248091648 bytes 2024-09-17T23:24:19.973 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:600 IOs, 327801856 bytes 2024-09-17T23:24:20.112 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:700 IOs, 380798976 bytes 2024-09-17T23:24:20.279 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:800 IOs, 464800768 bytes 2024-09-17T23:24:20.383 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:900 IOs, 491762688 bytes 2024-09-17T23:24:20.483 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1000 IOs, 510133248 bytes 2024-09-17T23:24:20.620 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1100 IOs, 585345024 bytes 2024-09-17T23:24:20.770 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1200 IOs, 676688896 bytes 2024-09-17T23:24:20.890 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1300 IOs, 683082752 bytes 2024-09-17T23:24:20.999 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1400 IOs, 712408064 bytes 2024-09-17T23:24:21.122 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1500 IOs, 730520576 bytes 2024-09-17T23:24:21.246 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1600 IOs, 779002880 bytes 2024-09-17T23:24:21.451 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1700 IOs, 896302080 bytes 2024-09-17T23:24:21.534 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1800 IOs, 896921600 bytes 2024-09-17T23:24:21.626 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1900 IOs, 918618112 bytes 2024-09-17T23:24:21.779 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2000 IOs, 989061120 bytes 2024-09-17T23:24:21.876 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2100 IOs, 1006994432 bytes 2024-09-17T23:24:21.980 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2200 IOs, 1031368704 bytes 2024-09-17T23:24:22.119 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2300 IOs, 1087191040 bytes 2024-09-17T23:24:22.218 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2400 IOs, 1098514432 bytes 2024-09-17T23:24:22.337 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2500 IOs, 1137387520 bytes 2024-09-17T23:24:22.441 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2600 IOs, 1159442432 bytes 2024-09-17T23:24:22.545 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2700 IOs, 1180820480 bytes 2024-09-17T23:24:22.662 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2800 IOs, 1217369088 bytes 2024-09-17T23:24:22.797 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2900 IOs, 1273320448 bytes 2024-09-17T23:24:22.901 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3000 IOs, 1295555584 bytes 2024-09-17T23:24:23.027 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3100 IOs, 1363531776 bytes 2024-09-17T23:24:23.114 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3200 IOs, 1365886976 bytes 2024-09-17T23:24:23.265 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3300 IOs, 1431031808 bytes 2024-09-17T23:24:23.385 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3400 IOs, 1471105024 bytes 2024-09-17T23:24:23.530 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3500 IOs, 1525251072 bytes 2024-09-17T23:24:23.660 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3600 IOs, 1587515392 bytes 2024-09-17T23:24:23.875 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3700 IOs, 1709384704 bytes 2024-09-17T23:24:23.995 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3800 IOs, 1748017152 bytes 2024-09-17T23:24:24.104 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ error_code=124 2024-09-17T23:24:24.105 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set -e 2024-09-17T23:24:24.105 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' 124 -eq 124 ']' 2024-09-17T23:24:24.105 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:24:24.105 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:24:24.105 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_48 2024-09-17T23:24:24.105 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_image_and_enable_mirror cluster2 mirror image_48 snapshot 128M 2024-09-17T23:24:24.105 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:24:24.105 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:24:24.105 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:24:24.105 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:24:24.105 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_48 2024-09-17T23:24:24.105 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:24:24.106 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=snapshot 2024-09-17T23:24:24.106 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n snapshot ']' 2024-09-17T23:24:24.106 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:24:24.106 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_image cluster2 mirror image_48 128M 2024-09-17T23:24:24.106 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:24:24.106 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:24:24.106 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:24:24.106 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:24:24.106 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_48 2024-09-17T23:24:24.106 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:24:24.106 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local size=128 2024-09-17T23:24:24.106 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n 128M ']' 2024-09-17T23:24:24.106 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ size=128M 2024-09-17T23:24:24.106 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:24:24.106 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_48 2024-09-17T23:24:24.155 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ is_pool_mirror_mode_image mirror 2024-09-17T23:24:24.156 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:24:24.156 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' image = image ']' 2024-09-17T23:24:24.156 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:24:24.156 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ enable_mirror cluster2 mirror image_48 snapshot 2024-09-17T23:24:24.156 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:24:24.156 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:24:24.156 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_48 2024-09-17T23:24:24.156 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=snapshot 2024-09-17T23:24:24.156 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_48 snapshot 2024-09-17T23:24:25.048 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Mirroring enabled 2024-09-17T23:24:25.060 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 info mirror/image_48 2024-09-17T23:24:25.101 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'image_48': 2024-09-17T23:24:25.101 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 128 MiB in 32 objects 2024-09-17T23:24:25.101 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:24:25.101 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 1 2024-09-17T23:24:25.101 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 14d945fc3449 2024-09-17T23:24:25.101 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.14d945fc3449 2024-09-17T23:24:25.101 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:24:25.101 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-17T23:24:25.102 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:24:25.102 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:24:25.102 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:24:24 2024 2024-09-17T23:24:25.102 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:24:24 2024 2024-09-17T23:24:25.102 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:24:24 2024 2024-09-17T23:24:25.102 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:24:25.102 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:24:25.102 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: 38a2f93b-1926-44fa-95d5-32067464b867 2024-09-17T23:24:25.102 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: true 2024-09-17T23:24:25.104 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n '' ']' 2024-09-17T23:24:25.104 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ stress_write_image cluster2 mirror image_48 2024-09-17T23:24:25.105 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:24:25.105 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:24:25.105 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_48 2024-09-17T23:24:25.105 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-17T23:24:25.106 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local duration=13 2024-09-17T23:24:25.106 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set +e 2024-09-17T23:24:25.106 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_48 --debug-rbd=20 --debug-journaler=20 2024-09-17T23:24:25.330 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:100 IOs, 51618816 bytes 2024-09-17T23:24:25.434 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:200 IOs, 74127360 bytes 2024-09-17T23:24:25.520 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:300 IOs, 82361344 bytes 2024-09-17T23:24:25.635 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:400 IOs, 139244544 bytes 2024-09-17T23:24:25.765 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:500 IOs, 166312960 bytes 2024-09-17T23:24:25.869 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:600 IOs, 189193216 bytes 2024-09-17T23:24:26.013 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:700 IOs, 254202880 bytes 2024-09-17T23:24:26.133 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:800 IOs, 292801536 bytes 2024-09-17T23:24:26.272 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:900 IOs, 345893888 bytes 2024-09-17T23:24:26.379 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1000 IOs, 374126592 bytes 2024-09-17T23:24:26.519 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1100 IOs, 435408896 bytes 2024-09-17T23:24:26.716 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1200 IOs, 556027904 bytes 2024-09-17T23:24:26.824 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1300 IOs, 580791296 bytes 2024-09-17T23:24:26.839 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2024-09-17T23:24:26.839 INFO:teuthology.orchestra.run:waiting for 600 2024-09-17T23:24:26.839 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2024-09-17T23:24:26.839 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2024-09-17T23:24:26.840 DEBUG:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2024-09-17T23:24:26.843 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2024-09-17T23:24:26.843 INFO:tasks.rbd_mirror_thrash:waiting for 102 secs before thrashing 2024-09-17T23:24:26.972 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1400 IOs, 642140160 bytes 2024-09-17T23:24:27.137 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1500 IOs, 687588352 bytes 2024-09-17T23:24:27.355 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1600 IOs, 823093248 bytes 2024-09-17T23:24:27.523 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1700 IOs, 904221696 bytes 2024-09-17T23:24:27.626 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1800 IOs, 914876416 bytes 2024-09-17T23:24:27.744 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1900 IOs, 955990016 bytes 2024-09-17T23:24:27.892 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2000 IOs, 1015500800 bytes 2024-09-17T23:24:28.013 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2100 IOs, 1059887104 bytes 2024-09-17T23:24:28.176 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2200 IOs, 1139779584 bytes 2024-09-17T23:24:28.293 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2300 IOs, 1175827456 bytes 2024-09-17T23:24:28.410 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2400 IOs, 1207485440 bytes 2024-09-17T23:24:28.504 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2500 IOs, 1219440640 bytes 2024-09-17T23:24:28.646 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2600 IOs, 1284136960 bytes 2024-09-17T23:24:28.785 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2700 IOs, 1342646272 bytes 2024-09-17T23:24:28.930 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2800 IOs, 1406373888 bytes 2024-09-17T23:24:29.013 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2900 IOs, 1408650240 bytes 2024-09-17T23:24:29.132 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3000 IOs, 1457575936 bytes 2024-09-17T23:24:29.267 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3100 IOs, 1499095040 bytes 2024-09-17T23:24:29.413 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3200 IOs, 1556833280 bytes 2024-09-17T23:24:29.538 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3300 IOs, 1619008512 bytes 2024-09-17T23:24:29.670 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3400 IOs, 1650957312 bytes 2024-09-17T23:24:29.793 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3500 IOs, 1691878400 bytes 2024-09-17T23:24:29.889 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3600 IOs, 1707172864 bytes 2024-09-17T23:24:30.023 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3700 IOs, 1760284672 bytes 2024-09-17T23:24:30.135 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3800 IOs, 1789280256 bytes 2024-09-17T23:24:30.278 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3900 IOs, 1842571264 bytes 2024-09-17T23:24:30.396 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4000 IOs, 1882475520 bytes 2024-09-17T23:24:30.507 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4100 IOs, 1921537024 bytes 2024-09-17T23:24:30.665 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4200 IOs, 1986441216 bytes 2024-09-17T23:24:30.747 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4300 IOs, 1987045376 bytes 2024-09-17T23:24:30.852 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4400 IOs, 2026310656 bytes 2024-09-17T23:24:30.997 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4500 IOs, 2072321024 bytes 2024-09-17T23:24:31.105 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4600 IOs, 2115731456 bytes 2024-09-17T23:24:31.223 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4700 IOs, 2128317440 bytes 2024-09-17T23:24:31.312 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4800 IOs, 2164852736 bytes 2024-09-17T23:24:31.482 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4900 IOs, 2226010112 bytes 2024-09-17T23:24:31.613 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5000 IOs, 2255910912 bytes 2024-09-17T23:24:31.725 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5100 IOs, 2292571136 bytes 2024-09-17T23:24:31.836 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5200 IOs, 2312089600 bytes 2024-09-17T23:24:32.006 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5300 IOs, 2399324160 bytes 2024-09-17T23:24:32.105 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5400 IOs, 2416485376 bytes 2024-09-17T23:24:32.302 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5500 IOs, 2530660352 bytes 2024-09-17T23:24:32.460 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5600 IOs, 2611773440 bytes 2024-09-17T23:24:32.617 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5700 IOs, 2679626752 bytes 2024-09-17T23:24:32.744 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5800 IOs, 2726486016 bytes 2024-09-17T23:24:32.862 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5900 IOs, 2761190400 bytes 2024-09-17T23:24:32.957 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6000 IOs, 2772990976 bytes 2024-09-17T23:24:33.048 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6100 IOs, 2780305408 bytes 2024-09-17T23:24:33.153 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6200 IOs, 2796581888 bytes 2024-09-17T23:24:33.256 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6300 IOs, 2815628288 bytes 2024-09-17T23:24:33.368 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6400 IOs, 2843103232 bytes 2024-09-17T23:24:33.459 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6500 IOs, 2852803584 bytes 2024-09-17T23:24:33.600 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6600 IOs, 2915351552 bytes 2024-09-17T23:24:33.722 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6700 IOs, 2944693248 bytes 2024-09-17T23:24:33.849 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6800 IOs, 3010265088 bytes 2024-09-17T23:24:34.066 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6900 IOs, 3054311424 bytes 2024-09-17T23:24:34.173 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7000 IOs, 3079187456 bytes 2024-09-17T23:24:34.265 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7100 IOs, 3089625088 bytes 2024-09-17T23:24:34.408 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7200 IOs, 3144408064 bytes 2024-09-17T23:24:34.577 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7300 IOs, 3224365056 bytes 2024-09-17T23:24:34.770 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7400 IOs, 3285348352 bytes 2024-09-17T23:24:34.884 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7500 IOs, 3325117440 bytes 2024-09-17T23:24:34.989 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7600 IOs, 3349952512 bytes 2024-09-17T23:24:35.101 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7700 IOs, 3375613952 bytes 2024-09-17T23:24:35.196 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7800 IOs, 3403691008 bytes 2024-09-17T23:24:35.365 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7900 IOs, 3475742720 bytes 2024-09-17T23:24:35.451 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8000 IOs, 3483406336 bytes 2024-09-17T23:24:35.590 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8100 IOs, 3539079168 bytes 2024-09-17T23:24:35.699 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8200 IOs, 3568798720 bytes 2024-09-17T23:24:35.809 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8300 IOs, 3597851648 bytes 2024-09-17T23:24:35.939 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8400 IOs, 3648434176 bytes 2024-09-17T23:24:36.066 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8500 IOs, 3700620288 bytes 2024-09-17T23:24:36.233 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8600 IOs, 3772336128 bytes 2024-09-17T23:24:36.363 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8700 IOs, 3826415616 bytes 2024-09-17T23:24:36.513 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8800 IOs, 3898628096 bytes 2024-09-17T23:24:36.602 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8900 IOs, 3899236352 bytes 2024-09-17T23:24:36.715 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9000 IOs, 3929555968 bytes 2024-09-17T23:24:36.829 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9100 IOs, 3969170432 bytes 2024-09-17T23:24:36.948 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9200 IOs, 4005587968 bytes 2024-09-17T23:24:37.059 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9300 IOs, 4050927616 bytes 2024-09-17T23:24:37.258 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9400 IOs, 4153647104 bytes 2024-09-17T23:24:37.408 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9500 IOs, 4220690432 bytes 2024-09-17T23:24:37.524 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9600 IOs, 4257135616 bytes 2024-09-17T23:24:37.669 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9700 IOs, 4317739008 bytes 2024-09-17T23:24:37.809 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9800 IOs, 4378652672 bytes 2024-09-17T23:24:37.938 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9900 IOs, 4426553344 bytes 2024-09-17T23:24:38.094 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10000 IOs, 4508617728 bytes 2024-09-17T23:24:38.112 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ error_code=124 2024-09-17T23:24:38.112 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set -e 2024-09-17T23:24:38.113 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' 124 -eq 124 ']' 2024-09-17T23:24:38.113 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:24:38.113 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:24:38.113 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_49 2024-09-17T23:24:38.113 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_image_and_enable_mirror cluster2 mirror image_49 snapshot 128M 2024-09-17T23:24:38.113 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:24:38.113 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:24:38.113 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:24:38.113 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:24:38.113 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_49 2024-09-17T23:24:38.113 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:24:38.113 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=snapshot 2024-09-17T23:24:38.113 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n snapshot ']' 2024-09-17T23:24:38.114 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:24:38.114 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_image cluster2 mirror image_49 128M 2024-09-17T23:24:38.114 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:24:38.114 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:24:38.114 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:24:38.114 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:24:38.115 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_49 2024-09-17T23:24:38.115 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:24:38.115 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local size=128 2024-09-17T23:24:38.115 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n 128M ']' 2024-09-17T23:24:38.115 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ size=128M 2024-09-17T23:24:38.115 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:24:38.115 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_49 2024-09-17T23:24:38.164 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ is_pool_mirror_mode_image mirror 2024-09-17T23:24:38.164 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:24:38.164 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' image = image ']' 2024-09-17T23:24:38.164 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:24:38.164 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ enable_mirror cluster2 mirror image_49 snapshot 2024-09-17T23:24:38.164 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:24:38.164 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:24:38.164 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_49 2024-09-17T23:24:38.165 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=snapshot 2024-09-17T23:24:38.165 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_49 snapshot 2024-09-17T23:24:39.067 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Mirroring enabled 2024-09-17T23:24:39.081 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 info mirror/image_49 2024-09-17T23:24:39.121 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'image_49': 2024-09-17T23:24:39.121 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 128 MiB in 32 objects 2024-09-17T23:24:39.121 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:24:39.121 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 1 2024-09-17T23:24:39.121 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 14e6eeebfc94 2024-09-17T23:24:39.121 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.14e6eeebfc94 2024-09-17T23:24:39.121 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:24:39.121 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-17T23:24:39.121 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:24:39.121 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:24:39.121 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:24:38 2024 2024-09-17T23:24:39.121 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:24:38 2024 2024-09-17T23:24:39.122 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:24:38 2024 2024-09-17T23:24:39.122 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:24:39.122 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:24:39.122 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: 61b5b6dd-6f04-4689-abba-edcd8570ecec 2024-09-17T23:24:39.122 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: true 2024-09-17T23:24:39.124 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n '' ']' 2024-09-17T23:24:39.124 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ stress_write_image cluster2 mirror image_49 2024-09-17T23:24:39.124 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:24:39.125 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:24:39.125 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_49 2024-09-17T23:24:39.125 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-17T23:24:39.126 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local duration=9 2024-09-17T23:24:39.127 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set +e 2024-09-17T23:24:39.127 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_49 --debug-rbd=20 --debug-journaler=20 2024-09-17T23:24:39.320 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:100 IOs, 19162112 bytes 2024-09-17T23:24:39.477 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:200 IOs, 119890944 bytes 2024-09-17T23:24:39.644 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:300 IOs, 207971328 bytes 2024-09-17T23:24:39.784 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:400 IOs, 258203648 bytes 2024-09-17T23:24:39.919 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:500 IOs, 311020544 bytes 2024-09-17T23:24:40.018 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:600 IOs, 330594304 bytes 2024-09-17T23:24:40.125 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:700 IOs, 370262016 bytes 2024-09-17T23:24:40.259 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:800 IOs, 412494848 bytes 2024-09-17T23:24:40.350 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:900 IOs, 437644288 bytes 2024-09-17T23:24:40.494 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1000 IOs, 490928128 bytes 2024-09-17T23:24:40.599 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1100 IOs, 491741184 bytes 2024-09-17T23:24:40.731 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1200 IOs, 542995456 bytes 2024-09-17T23:24:40.843 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1300 IOs, 582670336 bytes 2024-09-17T23:24:40.936 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1400 IOs, 583285760 bytes 2024-09-17T23:24:41.030 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1500 IOs, 592815104 bytes 2024-09-17T23:24:41.155 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1600 IOs, 630200320 bytes 2024-09-17T23:24:41.232 INFO:tasks.ceph.cluster2.osd.2.smithi038.stderr:2024-09-17T23:24:41.229+0000 7f1c4b04a640 -1 reset not still connected to 0x561cea0d0820 2024-09-17T23:24:41.258 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1700 IOs, 653817856 bytes 2024-09-17T23:24:41.383 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1800 IOs, 696333312 bytes 2024-09-17T23:24:41.514 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1900 IOs, 738323456 bytes 2024-09-17T23:24:41.639 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2000 IOs, 776472576 bytes 2024-09-17T23:24:41.730 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2100 IOs, 786675712 bytes 2024-09-17T23:24:41.815 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2200 IOs, 788587520 bytes 2024-09-17T23:24:41.916 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2300 IOs, 809737216 bytes 2024-09-17T23:24:42.001 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2400 IOs, 810357760 bytes 2024-09-17T23:24:42.121 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2500 IOs, 843879424 bytes 2024-09-17T23:24:42.395 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2600 IOs, 844485632 bytes 2024-09-17T23:24:42.504 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2700 IOs, 882495488 bytes 2024-09-17T23:24:42.708 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2800 IOs, 989698048 bytes 2024-09-17T23:24:42.813 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2900 IOs, 1012579328 bytes 2024-09-17T23:24:42.962 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3000 IOs, 1083641856 bytes 2024-09-17T23:24:43.125 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3100 IOs, 1167600640 bytes 2024-09-17T23:24:43.288 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3200 IOs, 1246486528 bytes 2024-09-17T23:24:43.398 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3300 IOs, 1280367616 bytes 2024-09-17T23:24:43.561 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3400 IOs, 1353972736 bytes 2024-09-17T23:24:43.704 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3500 IOs, 1418035200 bytes 2024-09-17T23:24:43.837 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3600 IOs, 1465494528 bytes 2024-09-17T23:24:43.969 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3700 IOs, 1526028288 bytes 2024-09-17T23:24:44.109 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3800 IOs, 1580163072 bytes 2024-09-17T23:24:44.273 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3900 IOs, 1672200192 bytes 2024-09-17T23:24:44.412 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4000 IOs, 1729609728 bytes 2024-09-17T23:24:44.511 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4100 IOs, 1745028096 bytes 2024-09-17T23:24:44.666 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4200 IOs, 1838268416 bytes 2024-09-17T23:24:44.754 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4300 IOs, 1842952192 bytes 2024-09-17T23:24:44.898 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4400 IOs, 1908766720 bytes 2024-09-17T23:24:45.013 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4500 IOs, 1934600192 bytes 2024-09-17T23:24:45.184 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4600 IOs, 2026559488 bytes 2024-09-17T23:24:45.292 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4700 IOs, 2041596928 bytes 2024-09-17T23:24:45.423 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4800 IOs, 2089334784 bytes 2024-09-17T23:24:45.540 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4900 IOs, 2118180864 bytes 2024-09-17T23:24:45.644 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5000 IOs, 2134362112 bytes 2024-09-17T23:24:45.742 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5100 IOs, 2145185792 bytes 2024-09-17T23:24:45.895 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5200 IOs, 2206931968 bytes 2024-09-17T23:24:46.083 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5300 IOs, 2309056512 bytes 2024-09-17T23:24:46.202 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5400 IOs, 2343297024 bytes 2024-09-17T23:24:46.364 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5500 IOs, 2422489088 bytes 2024-09-17T23:24:46.517 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5600 IOs, 2492109824 bytes 2024-09-17T23:24:46.663 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5700 IOs, 2554546176 bytes 2024-09-17T23:24:46.769 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5800 IOs, 2576763904 bytes 2024-09-17T23:24:46.894 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5900 IOs, 2615503872 bytes 2024-09-17T23:24:47.061 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6000 IOs, 2704791552 bytes 2024-09-17T23:24:47.170 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6100 IOs, 2729112576 bytes 2024-09-17T23:24:47.303 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6200 IOs, 2787332096 bytes 2024-09-17T23:24:47.451 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6300 IOs, 2846623744 bytes 2024-09-17T23:24:47.575 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6400 IOs, 2882690048 bytes 2024-09-17T23:24:47.745 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6500 IOs, 2899093504 bytes 2024-09-17T23:24:47.772 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6600 IOs, 2909015040 bytes 2024-09-17T23:24:47.892 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6700 IOs, 2961368064 bytes 2024-09-17T23:24:48.061 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6800 IOs, 3029351424 bytes 2024-09-17T23:24:48.133 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ error_code=124 2024-09-17T23:24:48.133 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set -e 2024-09-17T23:24:48.133 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' 124 -eq 124 ']' 2024-09-17T23:24:48.133 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:24:48.133 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:24:48.133 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_50 2024-09-17T23:24:48.133 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_image_and_enable_mirror cluster2 mirror image_50 snapshot 128M 2024-09-17T23:24:48.133 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:24:48.133 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:24:48.133 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:24:48.134 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:24:48.134 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_50 2024-09-17T23:24:48.134 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:24:48.134 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=snapshot 2024-09-17T23:24:48.134 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n snapshot ']' 2024-09-17T23:24:48.134 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:24:48.134 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_image cluster2 mirror image_50 128M 2024-09-17T23:24:48.134 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:24:48.134 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:24:48.134 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:24:48.135 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:24:48.135 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_50 2024-09-17T23:24:48.135 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:24:48.135 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local size=128 2024-09-17T23:24:48.135 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n 128M ']' 2024-09-17T23:24:48.135 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ size=128M 2024-09-17T23:24:48.135 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:24:48.135 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_50 2024-09-17T23:24:48.184 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ is_pool_mirror_mode_image mirror 2024-09-17T23:24:48.184 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:24:48.184 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' image = image ']' 2024-09-17T23:24:48.184 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:24:48.184 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ enable_mirror cluster2 mirror image_50 snapshot 2024-09-17T23:24:48.184 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:24:48.184 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:24:48.184 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_50 2024-09-17T23:24:48.184 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=snapshot 2024-09-17T23:24:48.185 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_50 snapshot 2024-09-17T23:24:49.087 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Mirroring enabled 2024-09-17T23:24:49.100 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 info mirror/image_50 2024-09-17T23:24:49.141 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'image_50': 2024-09-17T23:24:49.141 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 128 MiB in 32 objects 2024-09-17T23:24:49.141 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:24:49.141 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 1 2024-09-17T23:24:49.141 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 14f2d7b96992 2024-09-17T23:24:49.141 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.14f2d7b96992 2024-09-17T23:24:49.141 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:24:49.141 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-17T23:24:49.141 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:24:49.141 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:24:49.141 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:24:48 2024 2024-09-17T23:24:49.141 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:24:48 2024 2024-09-17T23:24:49.141 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:24:48 2024 2024-09-17T23:24:49.142 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:24:49.142 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:24:49.142 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: 865bd16a-a0e6-4a3a-bb53-8569e6a216ea 2024-09-17T23:24:49.142 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: true 2024-09-17T23:24:49.144 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n '' ']' 2024-09-17T23:24:49.144 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ stress_write_image cluster2 mirror image_50 2024-09-17T23:24:49.145 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:24:49.145 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:24:49.145 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_50 2024-09-17T23:24:49.145 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-17T23:24:49.146 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local duration=14 2024-09-17T23:24:49.146 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set +e 2024-09-17T23:24:49.146 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_50 --debug-rbd=20 --debug-journaler=20 2024-09-17T23:24:49.342 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:100 IOs, 24326144 bytes 2024-09-17T23:24:49.466 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:200 IOs, 74423296 bytes 2024-09-17T23:24:49.595 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:300 IOs, 128211968 bytes 2024-09-17T23:24:49.699 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:400 IOs, 145265664 bytes 2024-09-17T23:24:49.807 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:500 IOs, 171621376 bytes 2024-09-17T23:24:49.914 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:600 IOs, 194276352 bytes 2024-09-17T23:24:50.011 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:700 IOs, 211169280 bytes 2024-09-17T23:24:50.190 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:800 IOs, 309827584 bytes 2024-09-17T23:24:50.281 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:900 IOs, 315742208 bytes 2024-09-17T23:24:50.460 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1000 IOs, 426371072 bytes 2024-09-17T23:24:50.573 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1100 IOs, 468850688 bytes 2024-09-17T23:24:50.710 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1200 IOs, 510251008 bytes 2024-09-17T23:24:50.832 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1300 IOs, 547089408 bytes 2024-09-17T23:24:50.917 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1400 IOs, 547691520 bytes 2024-09-17T23:24:51.019 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1500 IOs, 571493376 bytes 2024-09-17T23:24:51.143 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1600 IOs, 606727168 bytes 2024-09-17T23:24:51.254 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1700 IOs, 634976256 bytes 2024-09-17T23:24:51.425 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1800 IOs, 724972544 bytes 2024-09-17T23:24:51.549 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1900 IOs, 765034496 bytes 2024-09-17T23:24:51.659 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2000 IOs, 787439616 bytes 2024-09-17T23:24:51.755 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2100 IOs, 808083456 bytes 2024-09-17T23:24:51.893 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2200 IOs, 858824704 bytes 2024-09-17T23:24:52.008 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2300 IOs, 891737088 bytes 2024-09-17T23:24:52.136 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2400 IOs, 932761600 bytes 2024-09-17T23:24:52.281 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2500 IOs, 999872512 bytes 2024-09-17T23:24:52.418 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2600 IOs, 1056750592 bytes 2024-09-17T23:24:52.555 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2700 IOs, 1110990848 bytes 2024-09-17T23:24:52.703 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2800 IOs, 1173549056 bytes 2024-09-17T23:24:52.801 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2900 IOs, 1189195776 bytes 2024-09-17T23:24:52.909 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3000 IOs, 1212937216 bytes 2024-09-17T23:24:53.027 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3100 IOs, 1252042752 bytes 2024-09-17T23:24:53.147 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3200 IOs, 1292233728 bytes 2024-09-17T23:24:53.257 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3300 IOs, 1323320320 bytes 2024-09-17T23:24:53.368 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3400 IOs, 1344564224 bytes 2024-09-17T23:24:53.461 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3500 IOs, 1371405312 bytes 2024-09-17T23:24:53.598 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3600 IOs, 1426033664 bytes 2024-09-17T23:24:53.747 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3700 IOs, 1486276608 bytes 2024-09-17T23:24:53.872 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3800 IOs, 1522402304 bytes 2024-09-17T23:24:54.003 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3900 IOs, 1599646720 bytes 2024-09-17T23:24:54.157 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4000 IOs, 1648526336 bytes 2024-09-17T23:24:54.273 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4100 IOs, 1707538432 bytes 2024-09-17T23:24:54.398 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4200 IOs, 1734628352 bytes 2024-09-17T23:24:54.483 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4300 IOs, 1740972032 bytes 2024-09-17T23:24:54.589 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4400 IOs, 1761901568 bytes 2024-09-17T23:24:54.699 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4500 IOs, 1809996800 bytes 2024-09-17T23:24:54.868 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4600 IOs, 1874848768 bytes 2024-09-17T23:24:54.997 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4700 IOs, 1926342656 bytes 2024-09-17T23:24:55.084 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4800 IOs, 1926958080 bytes 2024-09-17T23:24:55.229 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4900 IOs, 1993799680 bytes 2024-09-17T23:24:55.387 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5000 IOs, 2078558208 bytes 2024-09-17T23:24:55.485 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5100 IOs, 2088837120 bytes 2024-09-17T23:24:55.644 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5200 IOs, 2170986496 bytes 2024-09-17T23:24:55.737 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5300 IOs, 2172628992 bytes 2024-09-17T23:24:55.868 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5400 IOs, 2237336576 bytes 2024-09-17T23:24:56.043 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5500 IOs, 2317214720 bytes 2024-09-17T23:24:56.207 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5600 IOs, 2402666496 bytes 2024-09-17T23:24:56.313 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5700 IOs, 2424884224 bytes 2024-09-17T23:24:56.437 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5800 IOs, 2466117632 bytes 2024-09-17T23:24:56.552 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5900 IOs, 2501130240 bytes 2024-09-17T23:24:56.693 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6000 IOs, 2562911232 bytes 2024-09-17T23:24:56.802 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6100 IOs, 2588237824 bytes 2024-09-17T23:24:56.969 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6200 IOs, 2650658816 bytes 2024-09-17T23:24:57.075 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6300 IOs, 2667685888 bytes 2024-09-17T23:24:57.187 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6400 IOs, 2694856704 bytes 2024-09-17T23:24:57.313 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6500 IOs, 2739757056 bytes 2024-09-17T23:24:57.421 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6600 IOs, 2766934016 bytes 2024-09-17T23:24:57.566 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6700 IOs, 2826236928 bytes 2024-09-17T23:24:57.689 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6800 IOs, 2868385792 bytes 2024-09-17T23:24:57.804 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6900 IOs, 2895330304 bytes 2024-09-17T23:24:57.969 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7000 IOs, 2982830080 bytes 2024-09-17T23:24:58.100 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7100 IOs, 3029825536 bytes 2024-09-17T23:24:58.195 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7200 IOs, 3043396608 bytes 2024-09-17T23:24:58.342 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7300 IOs, 3115956224 bytes 2024-09-17T23:24:58.468 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7400 IOs, 3154446336 bytes 2024-09-17T23:24:58.559 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7500 IOs, 3165958144 bytes 2024-09-17T23:24:58.699 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7600 IOs, 3218561024 bytes 2024-09-17T23:24:58.844 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7700 IOs, 3280877568 bytes 2024-09-17T23:24:58.984 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7800 IOs, 3341778944 bytes 2024-09-17T23:24:59.118 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7900 IOs, 3392237568 bytes 2024-09-17T23:24:59.227 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8000 IOs, 3423354880 bytes 2024-09-17T23:24:59.312 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8100 IOs, 3425105920 bytes 2024-09-17T23:24:59.416 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8200 IOs, 3455847424 bytes 2024-09-17T23:24:59.529 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8300 IOs, 3483503616 bytes 2024-09-17T23:24:59.617 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8400 IOs, 3488627712 bytes 2024-09-17T23:24:59.728 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8500 IOs, 3558563840 bytes 2024-09-17T23:24:59.878 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8600 IOs, 3586746368 bytes 2024-09-17T23:25:00.040 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8700 IOs, 3673783296 bytes 2024-09-17T23:25:00.183 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8800 IOs, 3733954560 bytes 2024-09-17T23:25:00.296 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8900 IOs, 3781001216 bytes 2024-09-17T23:25:00.431 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9000 IOs, 3830756352 bytes 2024-09-17T23:25:00.540 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9100 IOs, 3853534208 bytes 2024-09-17T23:25:00.649 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9200 IOs, 3882702848 bytes 2024-09-17T23:25:00.750 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9300 IOs, 3902832640 bytes 2024-09-17T23:25:00.900 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9400 IOs, 3972359168 bytes 2024-09-17T23:25:01.019 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9500 IOs, 4012388352 bytes 2024-09-17T23:25:01.108 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9600 IOs, 4013010944 bytes 2024-09-17T23:25:01.226 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9700 IOs, 4051123200 bytes 2024-09-17T23:25:01.334 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9800 IOs, 4077688832 bytes 2024-09-17T23:25:01.449 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9900 IOs, 4113157120 bytes 2024-09-17T23:25:01.601 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10000 IOs, 4183161856 bytes 2024-09-17T23:25:01.749 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10100 IOs, 4252682240 bytes 2024-09-17T23:25:01.872 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10200 IOs, 4293679104 bytes 2024-09-17T23:25:01.988 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10300 IOs, 4327057408 bytes 2024-09-17T23:25:02.087 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10400 IOs, 4345732096 bytes 2024-09-17T23:25:02.218 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10500 IOs, 4397097984 bytes 2024-09-17T23:25:02.320 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10600 IOs, 4417384448 bytes 2024-09-17T23:25:02.474 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10700 IOs, 4493966336 bytes 2024-09-17T23:25:02.567 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10800 IOs, 4506655744 bytes 2024-09-17T23:25:02.680 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10900 IOs, 4539640832 bytes 2024-09-17T23:25:02.804 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:11000 IOs, 4592697344 bytes 2024-09-17T23:25:02.985 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:11100 IOs, 4673867776 bytes 2024-09-17T23:25:03.076 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:11200 IOs, 4686362624 bytes 2024-09-17T23:25:03.152 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ error_code=124 2024-09-17T23:25:03.152 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set -e 2024-09-17T23:25:03.152 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' 124 -eq 124 ']' 2024-09-17T23:25:03.152 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:25:03.153 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_pool_images cluster2 mirror 50 2024-09-17T23:25:03.153 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:25:03.153 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:25:03.153 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_count=50 2024-09-17T23:25:03.153 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:25:03.153 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local count 2024-09-17T23:25:03.153 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local last_count=0 2024-09-17T23:25:03.153 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:25:03.153 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ seq 1 40 2024-09-17T23:25:03.154 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in `seq 1 40` 2024-09-17T23:25:03.154 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 1 -ne 1 2024-09-17T23:25:03.154 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster cluster2 -p mirror mirror pool status 2024-09-17T23:25:03.154 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ grep 'images: ' 2024-09-17T23:25:03.154 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cut '-d ' -f 2 2024-09-17T23:25:03.189 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: failed to query services: (13) Permission denied 2024-09-17T23:25:03.193 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ count=50 2024-09-17T23:25:03.193 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 50 = 50 2024-09-17T23:25:03.193 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:25:03.193 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_pool_healthy cluster2 mirror 2024-09-17T23:25:03.193 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:25:03.193 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:25:03.193 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:25:03.193 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state 2024-09-17T23:25:03.193 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ seq 1 40 2024-09-17T23:25:03.194 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in `seq 1 40` 2024-09-17T23:25:03.194 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 1 -ne 1 2024-09-17T23:25:03.194 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster cluster2 -p mirror mirror pool status 2024-09-17T23:25:03.194 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ grep 'image health:' 2024-09-17T23:25:03.194 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cut '-d ' -f 3 2024-09-17T23:25:03.230 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: failed to query services: (13) Permission denied 2024-09-17T23:25:03.233 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ state=OK 2024-09-17T23:25:03.233 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test OK = ERROR 2024-09-17T23:25:03.234 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test OK = OK 2024-09-17T23:25:03.234 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:25:03.234 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_pool_images cluster1 mirror 50 2024-09-17T23:25:03.234 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:25:03.234 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:25:03.234 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_count=50 2024-09-17T23:25:03.234 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:25:03.234 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local count 2024-09-17T23:25:03.234 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local last_count=0 2024-09-17T23:25:03.234 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:25:03.234 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ seq 1 40 2024-09-17T23:25:03.235 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in `seq 1 40` 2024-09-17T23:25:03.235 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 1 -ne 1 2024-09-17T23:25:03.235 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2024-09-17T23:25:03.235 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ grep 'images: ' 2024-09-17T23:25:03.235 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cut '-d ' -f 2 2024-09-17T23:25:03.272 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: failed to query services: (13) Permission denied 2024-09-17T23:25:03.275 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ count=50 2024-09-17T23:25:03.275 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 50 = 50 2024-09-17T23:25:03.275 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:25:03.275 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_pool_healthy cluster1 mirror 2024-09-17T23:25:03.275 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:25:03.275 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:25:03.275 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:25:03.276 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state 2024-09-17T23:25:03.276 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ seq 1 40 2024-09-17T23:25:03.276 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in `seq 1 40` 2024-09-17T23:25:03.277 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 1 -ne 1 2024-09-17T23:25:03.277 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2024-09-17T23:25:03.277 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ grep 'image health:' 2024-09-17T23:25:03.277 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cut '-d ' -f 3 2024-09-17T23:25:03.314 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: failed to query services: (13) Permission denied 2024-09-17T23:25:03.317 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ state=OK 2024-09-17T23:25:03.318 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test OK = ERROR 2024-09-17T23:25:03.318 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test OK = OK 2024-09-17T23:25:03.318 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:25:03.318 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ testlog 'TEST: compare many images' 2024-09-17T23:25:03.318 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ tee -a /tmp/tmp.s5MEFDgsej/rbd-mirror.test.log 2024-09-17T23:25:03.318 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ date '+%F %T' 2024-09-17T23:25:03.318 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ echo 2024-09-17 23:25:03 TEST: compare many images 2024-09-17T23:25:03.319 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:2024-09-17 23:25:03 TEST: compare many images 2024-09-17T23:25:03.319 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ seq 1 50 2024-09-17T23:25:03.320 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:25:03.320 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_1 2024-09-17T23:25:03.320 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_snap cluster2 mirror image_1 snap 2024-09-17T23:25:03.320 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:25:03.320 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:25:03.321 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_1 2024-09-17T23:25:03.321 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:25:03.321 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror snap create image_1@snap --debug-rbd=20 --debug-journaler=20 2024-09-17T23:25:04.945 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.8082394949431762) > weight (0.25) 2024-09-17T23:25:04.946 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.4463593772418821) > weight (0.25) 2024-09-17T23:25:04.946 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.4528110166077862) > weight (0.25) 2024-09-17T23:25:04.946 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.7956457065577508) > weight (0.25) 2024-09-17T23:25:04.946 INFO:tasks.rbd_mirror_thrash:waiting for 87 secs before thrashing 2024-09-17T23:25:05.163 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_started cluster1 mirror image_1 2024-09-17T23:25:05.163 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:25:05.163 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:25:05.163 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_1 2024-09-17T23:25:05.164 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_state cluster1 mirror image_1 started 2024-09-17T23:25:05.164 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:25:05.164 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:25:05.164 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_1 2024-09-17T23:25:05.164 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=started 2024-09-17T23:25:05.164 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:25:05.164 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:25:05.164 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:25:06.165 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_replay_state cluster1 mirror image_1 started 2024-09-17T23:25:06.165 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:25:06.165 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:25:06.165 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_1 2024-09-17T23:25:06.165 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=started 2024-09-17T23:25:06.166 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_result 2024-09-17T23:25:06.166 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=stopped 2024-09-17T23:25:06.166 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_1 2024-09-17T23:25:06.166 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_instance=cluster1 2024-09-17T23:25:06.166 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:25:06.166 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ grep -i state 2024-09-17T23:25:06.166 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:25:06.166 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance=cluster1 2024-09-17T23:25:06.166 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local loop_instance 2024-09-17T23:25:06.166 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:25:06.166 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sleep 0 2024-09-17T23:25:06.168 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-17T23:25:06.168 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ seq 0 3 2024-09-17T23:25:06.168 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:25:06.168 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_1 2024-09-17T23:25:06.168 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:0 2024-09-17T23:25:06.169 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:25:06.169 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:25:06.169 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-17T23:25:06.169 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:0 2024-09-17T23:25:06.169 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:25:06.169 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:25:06.169 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:25:06.169 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=0 2024-09-17T23:25:06.169 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-17T23:25:06.169 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:25:06.169 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:25:06.169 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=0 2024-09-17T23:25:06.169 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:25:06.170 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-17T23:25:06.170 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:0 2024-09-17T23:25:06.170 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:25:06.170 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:25:06.170 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-17T23:25:06.170 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:0 2024-09-17T23:25:06.171 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:25:06.171 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:25:06.171 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:25:06.171 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:25:06.171 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-17T23:25:06.171 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:25:06.171 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:25:06.171 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=0 2024-09-17T23:25:06.172 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=0 2024-09-17T23:25:06.172 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-17T23:25:06.191 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:25:06.191 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:25:06.191 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:25:06.191 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_1 2024-09-17T23:25:06.315 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ return 0 2024-09-17T23:25:06.316 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ status_result=' "state": "Replaying"' 2024-09-17T23:25:06.316 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ echo ' "state": "Replaying"' 2024-09-17T23:25:06.316 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep -i Replaying 2024-09-17T23:25:06.317 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: "state": "Replaying" 2024-09-17T23:25:06.318 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_state=started 2024-09-17T23:25:06.318 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test started = started 2024-09-17T23:25:06.319 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:25:06.319 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_1 2024-09-17T23:25:06.319 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:25:06.319 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:25:06.319 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:25:06.319 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_1 2024-09-17T23:25:06.319 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = journal ']' 2024-09-17T23:25:06.319 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = snapshot ']' 2024-09-17T23:25:06.319 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_1 2024-09-17T23:25:06.319 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:25:06.319 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:25:06.319 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:25:06.319 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_1 2024-09-17T23:25:06.320 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster2-mirror-image_1.status 2024-09-17T23:25:06.320 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster2-mirror-image_1.status 2024-09-17T23:25:06.320 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:25:06.321 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_1.status 2024-09-17T23:25:06.321 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster1-mirror-image_1.status 2024-09-17T23:25:06.321 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster1-mirror-image_1.status 2024-09-17T23:25:06.321 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:25:06.321 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_status_log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_1.status 2024-09-17T23:25:06.321 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ mirror_image_snapshot cluster2 mirror image_1 2024-09-17T23:25:06.322 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:25:06.322 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:25:06.322 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_1 2024-09-17T23:25:06.322 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_1 2024-09-17T23:25:07.167 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Snapshot ID: 86 2024-09-17T23:25:07.180 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_1 /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_1.status 2024-09-17T23:25:07.180 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:25:07.180 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:25:07.180 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_1 2024-09-17T23:25:07.180 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_1.status 2024-09-17T23:25:07.180 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_1 --format xml 2024-09-17T23:25:07.181 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:25:07.228 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-17T23:25:07.231 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snapshot_id=86 2024-09-17T23:25:07.231 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:25:07.231 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:25:07.231 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.2 2024-09-17T23:25:07.432 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_1.status 2024-09-17T23:25:07.432 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:25:07.432 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:25:07.432 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_1 2024-09-17T23:25:07.433 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_1.status 2024-09-17T23:25:07.433 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2024-09-17T23:25:07.433 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:25:07.486 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:25:07.487 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:25:07.488 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:25:07.488 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:25:07.488 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:25:07.489 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:25:07.489 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 86 = '' 2024-09-17T23:25:07.489 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:25:07.489 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.4 2024-09-17T23:25:07.890 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_1.status 2024-09-17T23:25:07.890 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:25:07.890 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:25:07.891 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_1 2024-09-17T23:25:07.891 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_1.status 2024-09-17T23:25:07.891 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2024-09-17T23:25:07.891 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:25:07.946 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:25:07.946 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:25:07.948 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:25:07.948 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:25:07.948 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:25:07.948 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:25:07.948 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 86 = '' 2024-09-17T23:25:07.948 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:25:07.949 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.8 2024-09-17T23:25:08.750 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_1.status 2024-09-17T23:25:08.750 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:25:08.750 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:25:08.750 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_1 2024-09-17T23:25:08.750 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_1.status 2024-09-17T23:25:08.750 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2024-09-17T23:25:08.750 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:25:08.813 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:25:08.816 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id=86 2024-09-17T23:25:08.816 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 86 = 86 2024-09-17T23:25:08.816 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:25:08.816 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snap_present cluster1 mirror image_1 snap 2024-09-17T23:25:08.816 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:25:08.816 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:25:08.816 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_1 2024-09-17T23:25:08.816 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:25:08.816 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:25:08.816 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-17T23:25:08.816 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:25:09.816 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 info mirror/image_1@snap 2024-09-17T23:25:09.865 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'image_1': 2024-09-17T23:25:09.866 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 128 MiB in 32 objects 2024-09-17T23:25:09.866 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:25:09.866 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 2 2024-09-17T23:25:09.866 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 10c6a6a90160 2024-09-17T23:25:09.866 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.10c6a6a90160 2024-09-17T23:25:09.866 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:25:09.866 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-17T23:25:09.866 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:25:09.866 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:25:09.866 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:16:21 2024 2024-09-17T23:25:09.866 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:16:21 2024 2024-09-17T23:25:09.866 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:16:21 2024 2024-09-17T23:25:09.867 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: protected: False 2024-09-17T23:25:09.867 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:25:09.867 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:25:09.867 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: 5ca82a92-11a6-461d-96fa-d42ff94d4c65 2024-09-17T23:25:09.867 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: false 2024-09-17T23:25:09.870 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:25:09.870 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ compare_image_snaps mirror image_1 snap 2024-09-17T23:25:09.870 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:25:09.870 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_1 2024-09-17T23:25:09.870 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:25:09.870 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local ret=0 2024-09-17T23:25:09.871 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local rmt_export=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_1.export 2024-09-17T23:25:09.871 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local loc_export=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_1.export 2024-09-17T23:25:09.871 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_1.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_1.export 2024-09-17T23:25:09.871 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror export image_1@snap /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_1.export 2024-09-17T23:25:10.045 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-17T23:25:10.054 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 -p mirror export image_1@snap /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_1.export 2024-09-17T23:25:10.273 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-17T23:25:10.281 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ cmp /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_1.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_1.export 2024-09-17T23:25:10.343 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_1.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_1.export 2024-09-17T23:25:10.374 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:25:10.374 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:25:10.374 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_2 2024-09-17T23:25:10.374 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_snap cluster2 mirror image_2 snap 2024-09-17T23:25:10.374 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:25:10.374 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:25:10.374 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_2 2024-09-17T23:25:10.375 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:25:10.375 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror snap create image_2@snap --debug-rbd=20 --debug-journaler=20 2024-09-17T23:25:12.185 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_started cluster1 mirror image_2 2024-09-17T23:25:12.185 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:25:12.186 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:25:12.186 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_2 2024-09-17T23:25:12.186 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_state cluster1 mirror image_2 started 2024-09-17T23:25:12.186 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:25:12.186 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:25:12.186 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_2 2024-09-17T23:25:12.186 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=started 2024-09-17T23:25:12.186 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:25:12.186 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:25:12.186 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:25:13.186 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_replay_state cluster1 mirror image_2 started 2024-09-17T23:25:13.187 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:25:13.187 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:25:13.187 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_2 2024-09-17T23:25:13.187 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=started 2024-09-17T23:25:13.187 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_result 2024-09-17T23:25:13.187 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=stopped 2024-09-17T23:25:13.188 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_2 2024-09-17T23:25:13.188 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_instance=cluster1 2024-09-17T23:25:13.188 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:25:13.188 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ grep -i state 2024-09-17T23:25:13.188 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:25:13.188 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance=cluster1 2024-09-17T23:25:13.188 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local loop_instance 2024-09-17T23:25:13.188 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:25:13.189 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sleep 0 2024-09-17T23:25:13.189 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-17T23:25:13.189 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ seq 0 3 2024-09-17T23:25:13.191 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:25:13.192 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_2 2024-09-17T23:25:13.192 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:0 2024-09-17T23:25:13.192 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:25:13.192 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:25:13.192 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-17T23:25:13.192 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:0 2024-09-17T23:25:13.192 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:25:13.192 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:25:13.192 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:25:13.192 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=0 2024-09-17T23:25:13.192 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-17T23:25:13.193 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:25:13.193 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:25:13.193 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=0 2024-09-17T23:25:13.193 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:25:13.193 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-17T23:25:13.193 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:0 2024-09-17T23:25:13.193 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:25:13.193 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:25:13.193 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-17T23:25:13.193 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:0 2024-09-17T23:25:13.193 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:25:13.193 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:25:13.193 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:25:13.193 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:25:13.193 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-17T23:25:13.194 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:25:13.194 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:25:13.194 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=0 2024-09-17T23:25:13.194 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=0 2024-09-17T23:25:13.194 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-17T23:25:13.214 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:25:13.214 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:25:13.214 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:25:13.214 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_2 2024-09-17T23:25:13.341 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ return 0 2024-09-17T23:25:13.342 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ status_result=' "state": "Replaying"' 2024-09-17T23:25:13.342 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ echo ' "state": "Replaying"' 2024-09-17T23:25:13.342 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep -i Replaying 2024-09-17T23:25:13.343 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: "state": "Replaying" 2024-09-17T23:25:13.344 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_state=started 2024-09-17T23:25:13.344 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test started = started 2024-09-17T23:25:13.345 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:25:13.345 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_2 2024-09-17T23:25:13.345 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:25:13.345 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:25:13.345 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:25:13.345 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_2 2024-09-17T23:25:13.345 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = journal ']' 2024-09-17T23:25:13.345 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = snapshot ']' 2024-09-17T23:25:13.345 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_2 2024-09-17T23:25:13.345 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:25:13.345 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:25:13.345 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:25:13.345 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_2 2024-09-17T23:25:13.345 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster2-mirror-image_2.status 2024-09-17T23:25:13.346 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster2-mirror-image_2.status 2024-09-17T23:25:13.346 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:25:13.346 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_2.status 2024-09-17T23:25:13.346 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster1-mirror-image_2.status 2024-09-17T23:25:13.346 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster1-mirror-image_2.status 2024-09-17T23:25:13.346 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:25:13.347 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_status_log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_2.status 2024-09-17T23:25:13.347 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ mirror_image_snapshot cluster2 mirror image_2 2024-09-17T23:25:13.347 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:25:13.347 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:25:13.347 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_2 2024-09-17T23:25:13.348 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_2 2024-09-17T23:25:14.204 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Snapshot ID: 88 2024-09-17T23:25:14.216 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_2 /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_2.status 2024-09-17T23:25:14.216 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:25:14.216 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:25:14.216 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_2 2024-09-17T23:25:14.216 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_2.status 2024-09-17T23:25:14.217 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_2 --format xml 2024-09-17T23:25:14.217 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:25:14.264 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-17T23:25:14.266 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snapshot_id=88 2024-09-17T23:25:14.266 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:25:14.267 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:25:14.267 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.2 2024-09-17T23:25:14.468 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_2.status 2024-09-17T23:25:14.468 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:25:14.468 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:25:14.468 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_2 2024-09-17T23:25:14.468 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_2.status 2024-09-17T23:25:14.468 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2024-09-17T23:25:14.468 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:25:14.520 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:25:14.522 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id=36 2024-09-17T23:25:14.522 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 88 = 36 2024-09-17T23:25:14.522 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:25:14.522 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.4 2024-09-17T23:25:14.923 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_2.status 2024-09-17T23:25:14.923 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:25:14.923 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:25:14.924 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_2 2024-09-17T23:25:14.924 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_2.status 2024-09-17T23:25:14.924 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2024-09-17T23:25:14.924 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:25:14.980 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:25:14.981 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:25:14.982 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:25:14.982 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:25:14.982 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:25:14.983 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:25:14.983 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 88 = '' 2024-09-17T23:25:14.983 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:25:14.983 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.8 2024-09-17T23:25:15.785 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_2.status 2024-09-17T23:25:15.785 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:25:15.785 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:25:15.785 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_2 2024-09-17T23:25:15.785 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_2.status 2024-09-17T23:25:15.785 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2024-09-17T23:25:15.785 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:25:15.834 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:25:15.836 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id=88 2024-09-17T23:25:15.837 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 88 = 88 2024-09-17T23:25:15.837 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:25:15.837 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snap_present cluster1 mirror image_2 snap 2024-09-17T23:25:15.837 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:25:15.837 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:25:15.837 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_2 2024-09-17T23:25:15.837 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:25:15.837 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:25:15.837 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-17T23:25:15.837 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:25:16.838 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 info mirror/image_2@snap 2024-09-17T23:25:16.887 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'image_2': 2024-09-17T23:25:16.887 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 128 MiB in 32 objects 2024-09-17T23:25:16.888 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:25:16.888 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 2 2024-09-17T23:25:16.888 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 10c66efd7daa 2024-09-17T23:25:16.888 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.10c66efd7daa 2024-09-17T23:25:16.888 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:25:16.888 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-17T23:25:16.888 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:25:16.888 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:25:16.888 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:16:29 2024 2024-09-17T23:25:16.888 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:16:29 2024 2024-09-17T23:25:16.889 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:16:29 2024 2024-09-17T23:25:16.889 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: protected: False 2024-09-17T23:25:16.889 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:25:16.889 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:25:16.889 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: e4119d6f-e55a-46f9-97e1-502365b9e208 2024-09-17T23:25:16.889 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: false 2024-09-17T23:25:16.891 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:25:16.891 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ compare_image_snaps mirror image_2 snap 2024-09-17T23:25:16.892 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:25:16.892 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_2 2024-09-17T23:25:16.892 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:25:16.892 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local ret=0 2024-09-17T23:25:16.892 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local rmt_export=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_2.export 2024-09-17T23:25:16.892 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local loc_export=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_2.export 2024-09-17T23:25:16.892 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_2.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_2.export 2024-09-17T23:25:16.892 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror export image_2@snap /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_2.export 2024-09-17T23:25:17.067 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-17T23:25:17.074 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 -p mirror export image_2@snap /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_2.export 2024-09-17T23:25:17.306 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-17T23:25:17.313 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ cmp /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_2.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_2.export 2024-09-17T23:25:17.375 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_2.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_2.export 2024-09-17T23:25:17.407 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:25:17.407 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:25:17.407 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_3 2024-09-17T23:25:17.408 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_snap cluster2 mirror image_3 snap 2024-09-17T23:25:17.408 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:25:17.408 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:25:17.408 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_3 2024-09-17T23:25:17.408 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:25:17.408 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror snap create image_3@snap --debug-rbd=20 --debug-journaler=20 2024-09-17T23:25:19.236 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_started cluster1 mirror image_3 2024-09-17T23:25:19.236 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:25:19.236 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:25:19.237 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_3 2024-09-17T23:25:19.237 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_state cluster1 mirror image_3 started 2024-09-17T23:25:19.237 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:25:19.237 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:25:19.237 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_3 2024-09-17T23:25:19.237 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=started 2024-09-17T23:25:19.237 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:25:19.237 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:25:19.237 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:25:20.238 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_replay_state cluster1 mirror image_3 started 2024-09-17T23:25:20.238 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:25:20.238 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:25:20.238 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_3 2024-09-17T23:25:20.238 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=started 2024-09-17T23:25:20.238 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_result 2024-09-17T23:25:20.238 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=stopped 2024-09-17T23:25:20.239 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_3 2024-09-17T23:25:20.239 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_instance=cluster1 2024-09-17T23:25:20.239 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:25:20.239 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ grep -i state 2024-09-17T23:25:20.240 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:25:20.240 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance=cluster1 2024-09-17T23:25:20.240 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local loop_instance 2024-09-17T23:25:20.240 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:25:20.240 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sleep 0 2024-09-17T23:25:20.240 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-17T23:25:20.243 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ seq 0 3 2024-09-17T23:25:20.243 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:25:20.243 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_3 2024-09-17T23:25:20.243 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:0 2024-09-17T23:25:20.243 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:25:20.243 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:25:20.243 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-17T23:25:20.243 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:0 2024-09-17T23:25:20.243 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:25:20.243 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:25:20.243 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:25:20.243 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=0 2024-09-17T23:25:20.243 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-17T23:25:20.243 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:25:20.244 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:25:20.244 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=0 2024-09-17T23:25:20.244 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:25:20.244 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-17T23:25:20.244 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:0 2024-09-17T23:25:20.244 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:25:20.244 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:25:20.244 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-17T23:25:20.244 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:0 2024-09-17T23:25:20.244 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:25:20.244 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:25:20.244 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:25:20.244 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:25:20.244 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-17T23:25:20.245 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:25:20.245 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:25:20.245 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=0 2024-09-17T23:25:20.245 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=0 2024-09-17T23:25:20.245 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-17T23:25:20.265 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:25:20.265 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:25:20.265 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:25:20.265 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_3 2024-09-17T23:25:20.391 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ return 0 2024-09-17T23:25:20.392 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ status_result=' "state": "Replaying"' 2024-09-17T23:25:20.392 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ echo ' "state": "Replaying"' 2024-09-17T23:25:20.392 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep -i Replaying 2024-09-17T23:25:20.393 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: "state": "Replaying" 2024-09-17T23:25:20.394 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_state=started 2024-09-17T23:25:20.394 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test started = started 2024-09-17T23:25:20.395 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:25:20.395 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_3 2024-09-17T23:25:20.395 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:25:20.395 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:25:20.395 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:25:20.395 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_3 2024-09-17T23:25:20.395 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = journal ']' 2024-09-17T23:25:20.395 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = snapshot ']' 2024-09-17T23:25:20.395 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_3 2024-09-17T23:25:20.395 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:25:20.395 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:25:20.395 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:25:20.395 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_3 2024-09-17T23:25:20.395 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster2-mirror-image_3.status 2024-09-17T23:25:20.396 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster2-mirror-image_3.status 2024-09-17T23:25:20.396 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:25:20.396 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_3.status 2024-09-17T23:25:20.396 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster1-mirror-image_3.status 2024-09-17T23:25:20.396 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster1-mirror-image_3.status 2024-09-17T23:25:20.396 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:25:20.397 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_status_log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_3.status 2024-09-17T23:25:20.397 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ mirror_image_snapshot cluster2 mirror image_3 2024-09-17T23:25:20.397 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:25:20.397 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:25:20.397 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_3 2024-09-17T23:25:20.398 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_3 2024-09-17T23:25:21.233 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Snapshot ID: 90 2024-09-17T23:25:21.246 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_3 /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_3.status 2024-09-17T23:25:21.246 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:25:21.246 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:25:21.247 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_3 2024-09-17T23:25:21.247 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_3.status 2024-09-17T23:25:21.247 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_3 --format xml 2024-09-17T23:25:21.247 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:25:21.294 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-17T23:25:21.296 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snapshot_id=90 2024-09-17T23:25:21.296 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:25:21.296 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:25:21.296 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.2 2024-09-17T23:25:21.498 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_3.status 2024-09-17T23:25:21.498 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:25:21.498 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:25:21.498 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_3 2024-09-17T23:25:21.498 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_3.status 2024-09-17T23:25:21.498 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2024-09-17T23:25:21.498 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:25:21.549 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:25:21.551 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id=37 2024-09-17T23:25:21.551 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 90 = 37 2024-09-17T23:25:21.551 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:25:21.551 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.4 2024-09-17T23:25:21.952 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_3.status 2024-09-17T23:25:21.953 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:25:21.953 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:25:21.953 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_3 2024-09-17T23:25:21.953 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_3.status 2024-09-17T23:25:21.953 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2024-09-17T23:25:21.953 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:25:22.005 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:25:22.007 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id=37 2024-09-17T23:25:22.007 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 90 = 37 2024-09-17T23:25:22.007 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:25:22.007 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.8 2024-09-17T23:25:22.809 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_3.status 2024-09-17T23:25:22.809 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:25:22.809 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:25:22.809 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_3 2024-09-17T23:25:22.809 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_3.status 2024-09-17T23:25:22.809 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2024-09-17T23:25:22.809 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:25:22.862 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:25:22.863 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:25:22.864 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:25:22.864 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:25:22.864 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:25:22.865 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:25:22.865 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 90 = '' 2024-09-17T23:25:22.865 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:25:22.865 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1.6 2024-09-17T23:25:24.466 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_3.status 2024-09-17T23:25:24.467 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:25:24.467 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:25:24.467 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_3 2024-09-17T23:25:24.467 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_3.status 2024-09-17T23:25:24.467 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2024-09-17T23:25:24.467 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:25:24.521 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:25:24.524 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id=90 2024-09-17T23:25:24.524 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 90 = 90 2024-09-17T23:25:24.524 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:25:24.524 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snap_present cluster1 mirror image_3 snap 2024-09-17T23:25:24.524 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:25:24.524 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:25:24.524 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_3 2024-09-17T23:25:24.524 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:25:24.525 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:25:24.525 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-17T23:25:24.525 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:25:25.525 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 info mirror/image_3@snap 2024-09-17T23:25:25.574 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'image_3': 2024-09-17T23:25:25.574 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 128 MiB in 32 objects 2024-09-17T23:25:25.575 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:25:25.575 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 2 2024-09-17T23:25:25.575 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 10c6468b079f 2024-09-17T23:25:25.575 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.10c6468b079f 2024-09-17T23:25:25.575 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:25:25.575 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-17T23:25:25.575 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:25:25.575 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:25:25.575 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:16:44 2024 2024-09-17T23:25:25.575 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:16:44 2024 2024-09-17T23:25:25.575 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:16:44 2024 2024-09-17T23:25:25.575 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: protected: False 2024-09-17T23:25:25.575 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:25:25.575 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:25:25.575 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: cb923d84-7e5a-4d91-9543-3952a9f3edb7 2024-09-17T23:25:25.575 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: false 2024-09-17T23:25:25.578 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:25:25.578 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ compare_image_snaps mirror image_3 snap 2024-09-17T23:25:25.578 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:25:25.579 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_3 2024-09-17T23:25:25.579 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:25:25.579 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local ret=0 2024-09-17T23:25:25.579 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local rmt_export=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_3.export 2024-09-17T23:25:25.579 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local loc_export=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_3.export 2024-09-17T23:25:25.579 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_3.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_3.export 2024-09-17T23:25:25.579 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror export image_3@snap /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_3.export 2024-09-17T23:25:25.750 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-17T23:25:25.757 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 -p mirror export image_3@snap /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_3.export 2024-09-17T23:25:25.967 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-17T23:25:25.974 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ cmp /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_3.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_3.export 2024-09-17T23:25:26.036 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_3.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_3.export 2024-09-17T23:25:26.068 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:25:26.069 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:25:26.069 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_4 2024-09-17T23:25:26.069 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_snap cluster2 mirror image_4 snap 2024-09-17T23:25:26.069 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:25:26.069 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:25:26.069 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_4 2024-09-17T23:25:26.069 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:25:26.069 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror snap create image_4@snap --debug-rbd=20 --debug-journaler=20 2024-09-17T23:25:27.249 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_started cluster1 mirror image_4 2024-09-17T23:25:27.250 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:25:27.250 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:25:27.250 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_4 2024-09-17T23:25:27.250 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_state cluster1 mirror image_4 started 2024-09-17T23:25:27.250 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:25:27.250 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:25:27.250 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_4 2024-09-17T23:25:27.250 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=started 2024-09-17T23:25:27.250 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:25:27.250 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:25:27.250 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:25:28.251 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_replay_state cluster1 mirror image_4 started 2024-09-17T23:25:28.251 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:25:28.251 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:25:28.251 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_4 2024-09-17T23:25:28.251 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=started 2024-09-17T23:25:28.251 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_result 2024-09-17T23:25:28.251 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=stopped 2024-09-17T23:25:28.252 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_4 2024-09-17T23:25:28.252 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_instance=cluster1 2024-09-17T23:25:28.252 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:25:28.252 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ grep -i state 2024-09-17T23:25:28.253 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:25:28.253 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance=cluster1 2024-09-17T23:25:28.253 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local loop_instance 2024-09-17T23:25:28.253 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:25:28.253 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sleep 0 2024-09-17T23:25:28.253 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-17T23:25:28.253 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ seq 0 3 2024-09-17T23:25:28.255 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:25:28.255 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_4 2024-09-17T23:25:28.255 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:0 2024-09-17T23:25:28.255 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:25:28.255 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:25:28.255 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-17T23:25:28.255 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:0 2024-09-17T23:25:28.256 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:25:28.256 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:25:28.256 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:25:28.256 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=0 2024-09-17T23:25:28.256 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-17T23:25:28.256 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:25:28.256 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:25:28.256 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=0 2024-09-17T23:25:28.256 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:25:28.256 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-17T23:25:28.256 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:0 2024-09-17T23:25:28.256 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:25:28.256 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:25:28.256 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-17T23:25:28.257 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:0 2024-09-17T23:25:28.257 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:25:28.257 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:25:28.257 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:25:28.258 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:25:28.258 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-17T23:25:28.258 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:25:28.258 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:25:28.258 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=0 2024-09-17T23:25:28.258 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=0 2024-09-17T23:25:28.258 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-17T23:25:28.278 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:25:28.278 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:25:28.278 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:25:28.278 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_4 2024-09-17T23:25:28.407 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ return 0 2024-09-17T23:25:28.408 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ status_result=' "state": "Replaying"' 2024-09-17T23:25:28.408 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ echo ' "state": "Replaying"' 2024-09-17T23:25:28.408 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep -i Replaying 2024-09-17T23:25:28.409 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: "state": "Replaying" 2024-09-17T23:25:28.410 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_state=started 2024-09-17T23:25:28.410 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test started = started 2024-09-17T23:25:28.410 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:25:28.410 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_4 2024-09-17T23:25:28.410 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:25:28.410 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:25:28.411 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:25:28.411 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_4 2024-09-17T23:25:28.411 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = journal ']' 2024-09-17T23:25:28.411 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = snapshot ']' 2024-09-17T23:25:28.411 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_4 2024-09-17T23:25:28.411 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:25:28.411 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:25:28.411 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:25:28.411 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_4 2024-09-17T23:25:28.411 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster2-mirror-image_4.status 2024-09-17T23:25:28.411 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster2-mirror-image_4.status 2024-09-17T23:25:28.411 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:25:28.412 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_4.status 2024-09-17T23:25:28.412 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster1-mirror-image_4.status 2024-09-17T23:25:28.412 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster1-mirror-image_4.status 2024-09-17T23:25:28.412 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:25:28.413 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_status_log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_4.status 2024-09-17T23:25:28.413 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ mirror_image_snapshot cluster2 mirror image_4 2024-09-17T23:25:28.413 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:25:28.413 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:25:28.413 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_4 2024-09-17T23:25:28.413 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_4 2024-09-17T23:25:29.253 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Snapshot ID: 92 2024-09-17T23:25:29.265 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_4 /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_4.status 2024-09-17T23:25:29.266 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:25:29.266 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:25:29.266 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_4 2024-09-17T23:25:29.266 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_4.status 2024-09-17T23:25:29.266 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_4 --format xml 2024-09-17T23:25:29.266 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:25:29.312 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-17T23:25:29.314 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snapshot_id=92 2024-09-17T23:25:29.314 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:25:29.314 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:25:29.314 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.2 2024-09-17T23:25:29.516 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_4.status 2024-09-17T23:25:29.516 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:25:29.516 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:25:29.516 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_4 2024-09-17T23:25:29.516 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_4.status 2024-09-17T23:25:29.516 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2024-09-17T23:25:29.516 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:25:29.566 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:25:29.568 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id=38 2024-09-17T23:25:29.568 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 92 = 38 2024-09-17T23:25:29.568 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:25:29.568 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.4 2024-09-17T23:25:29.970 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_4.status 2024-09-17T23:25:29.970 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:25:29.970 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:25:29.970 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_4 2024-09-17T23:25:29.971 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_4.status 2024-09-17T23:25:29.971 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2024-09-17T23:25:29.971 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:25:30.224 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:25:30.225 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:25:30.226 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:25:30.226 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:25:30.226 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:25:30.227 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:25:30.227 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 92 = '' 2024-09-17T23:25:30.227 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:25:30.227 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.8 2024-09-17T23:25:31.028 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_4.status 2024-09-17T23:25:31.029 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:25:31.029 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:25:31.029 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_4 2024-09-17T23:25:31.029 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_4.status 2024-09-17T23:25:31.029 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2024-09-17T23:25:31.029 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:25:31.085 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:25:31.086 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:25:31.087 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:25:31.087 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:25:31.087 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:25:31.088 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:25:31.088 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 92 = '' 2024-09-17T23:25:31.088 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:25:31.088 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1.6 2024-09-17T23:25:32.689 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_4.status 2024-09-17T23:25:32.689 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:25:32.689 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:25:32.689 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_4 2024-09-17T23:25:32.689 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_4.status 2024-09-17T23:25:32.690 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2024-09-17T23:25:32.690 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:25:32.743 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:25:32.745 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id=92 2024-09-17T23:25:32.745 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 92 = 92 2024-09-17T23:25:32.746 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:25:32.746 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snap_present cluster1 mirror image_4 snap 2024-09-17T23:25:32.746 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:25:32.746 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:25:32.746 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_4 2024-09-17T23:25:32.746 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:25:32.746 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:25:32.746 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-17T23:25:32.746 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:25:33.747 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 info mirror/image_4@snap 2024-09-17T23:25:33.794 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'image_4': 2024-09-17T23:25:33.794 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 128 MiB in 32 objects 2024-09-17T23:25:33.794 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:25:33.794 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 2 2024-09-17T23:25:33.794 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 10c64facbebc 2024-09-17T23:25:33.794 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.10c64facbebc 2024-09-17T23:25:33.795 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:25:33.795 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-17T23:25:33.795 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:25:33.795 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:25:33.795 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:16:50 2024 2024-09-17T23:25:33.795 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:16:50 2024 2024-09-17T23:25:33.795 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:16:50 2024 2024-09-17T23:25:33.795 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: protected: False 2024-09-17T23:25:33.795 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:25:33.795 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:25:33.795 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: 1445a68d-ae17-48f5-9088-ad3e0f9e8951 2024-09-17T23:25:33.795 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: false 2024-09-17T23:25:33.798 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:25:33.798 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ compare_image_snaps mirror image_4 snap 2024-09-17T23:25:33.798 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:25:33.798 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_4 2024-09-17T23:25:33.799 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:25:33.799 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local ret=0 2024-09-17T23:25:33.799 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local rmt_export=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_4.export 2024-09-17T23:25:33.799 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local loc_export=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_4.export 2024-09-17T23:25:33.799 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_4.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_4.export 2024-09-17T23:25:33.799 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror export image_4@snap /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_4.export 2024-09-17T23:25:33.972 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-17T23:25:33.980 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 -p mirror export image_4@snap /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_4.export 2024-09-17T23:25:34.190 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-17T23:25:34.198 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ cmp /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_4.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_4.export 2024-09-17T23:25:34.259 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_4.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_4.export 2024-09-17T23:25:34.290 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:25:34.290 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:25:34.291 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_5 2024-09-17T23:25:34.291 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_snap cluster2 mirror image_5 snap 2024-09-17T23:25:34.291 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:25:34.291 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:25:34.291 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_5 2024-09-17T23:25:34.291 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:25:34.291 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror snap create image_5@snap --debug-rbd=20 --debug-journaler=20 2024-09-17T23:25:35.419 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_started cluster1 mirror image_5 2024-09-17T23:25:35.419 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:25:35.419 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:25:35.420 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_5 2024-09-17T23:25:35.420 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_state cluster1 mirror image_5 started 2024-09-17T23:25:35.420 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:25:35.420 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:25:35.420 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_5 2024-09-17T23:25:35.420 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=started 2024-09-17T23:25:35.420 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:25:35.420 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:25:35.420 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:25:36.421 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_replay_state cluster1 mirror image_5 started 2024-09-17T23:25:36.422 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:25:36.422 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:25:36.422 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_5 2024-09-17T23:25:36.422 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=started 2024-09-17T23:25:36.422 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_result 2024-09-17T23:25:36.422 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=stopped 2024-09-17T23:25:36.422 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_5 2024-09-17T23:25:36.422 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_instance=cluster1 2024-09-17T23:25:36.422 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:25:36.422 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ grep -i state 2024-09-17T23:25:36.422 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:25:36.422 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance=cluster1 2024-09-17T23:25:36.423 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local loop_instance 2024-09-17T23:25:36.423 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:25:36.423 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sleep 0 2024-09-17T23:25:36.425 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-17T23:25:36.425 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ seq 0 3 2024-09-17T23:25:36.425 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:25:36.426 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_5 2024-09-17T23:25:36.426 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:0 2024-09-17T23:25:36.426 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:25:36.426 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:25:36.426 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-17T23:25:36.426 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:0 2024-09-17T23:25:36.426 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:25:36.426 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:25:36.426 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:25:36.427 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=0 2024-09-17T23:25:36.427 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-17T23:25:36.427 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:25:36.427 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:25:36.427 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=0 2024-09-17T23:25:36.427 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:25:36.427 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-17T23:25:36.427 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:0 2024-09-17T23:25:36.427 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:25:36.427 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:25:36.427 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-17T23:25:36.427 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:0 2024-09-17T23:25:36.427 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:25:36.427 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:25:36.427 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:25:36.427 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:25:36.428 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-17T23:25:36.428 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:25:36.428 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:25:36.428 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=0 2024-09-17T23:25:36.428 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=0 2024-09-17T23:25:36.428 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-17T23:25:36.447 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:25:36.447 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:25:36.447 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:25:36.448 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_5 2024-09-17T23:25:36.576 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ return 0 2024-09-17T23:25:36.576 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ status_result=' "state": "Replaying"' 2024-09-17T23:25:36.576 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ echo ' "state": "Replaying"' 2024-09-17T23:25:36.577 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep -i Replaying 2024-09-17T23:25:36.577 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: "state": "Replaying" 2024-09-17T23:25:36.579 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_state=started 2024-09-17T23:25:36.579 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test started = started 2024-09-17T23:25:36.579 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:25:36.579 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_5 2024-09-17T23:25:36.579 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:25:36.579 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:25:36.579 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:25:36.579 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_5 2024-09-17T23:25:36.579 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = journal ']' 2024-09-17T23:25:36.580 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = snapshot ']' 2024-09-17T23:25:36.580 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_5 2024-09-17T23:25:36.580 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:25:36.580 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:25:36.580 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:25:36.580 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_5 2024-09-17T23:25:36.580 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster2-mirror-image_5.status 2024-09-17T23:25:36.580 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster2-mirror-image_5.status 2024-09-17T23:25:36.580 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:25:36.581 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_5.status 2024-09-17T23:25:36.582 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster1-mirror-image_5.status 2024-09-17T23:25:36.582 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster1-mirror-image_5.status 2024-09-17T23:25:36.582 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:25:36.582 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_status_log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_5.status 2024-09-17T23:25:36.582 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ mirror_image_snapshot cluster2 mirror image_5 2024-09-17T23:25:36.582 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:25:36.582 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:25:36.582 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_5 2024-09-17T23:25:36.582 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_5 2024-09-17T23:25:37.422 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Snapshot ID: 94 2024-09-17T23:25:37.435 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_5 /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_5.status 2024-09-17T23:25:37.435 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:25:37.435 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:25:37.435 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_5 2024-09-17T23:25:37.435 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_5.status 2024-09-17T23:25:37.435 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_5 --format xml 2024-09-17T23:25:37.435 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:25:37.483 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-17T23:25:37.485 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snapshot_id=94 2024-09-17T23:25:37.485 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:25:37.485 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:25:37.485 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.2 2024-09-17T23:25:37.687 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_5.status 2024-09-17T23:25:37.687 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:25:37.687 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:25:37.687 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_5 2024-09-17T23:25:37.687 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_5.status 2024-09-17T23:25:37.687 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2024-09-17T23:25:37.687 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:25:37.736 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:25:37.738 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id=39 2024-09-17T23:25:37.738 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 94 = 39 2024-09-17T23:25:37.738 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:25:37.738 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.4 2024-09-17T23:25:38.139 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_5.status 2024-09-17T23:25:38.140 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:25:38.140 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:25:38.140 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_5 2024-09-17T23:25:38.140 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_5.status 2024-09-17T23:25:38.140 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2024-09-17T23:25:38.140 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:25:38.190 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:25:38.192 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id=39 2024-09-17T23:25:38.192 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 94 = 39 2024-09-17T23:25:38.192 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:25:38.192 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.8 2024-09-17T23:25:38.993 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_5.status 2024-09-17T23:25:38.994 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:25:38.994 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:25:38.994 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_5 2024-09-17T23:25:38.994 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_5.status 2024-09-17T23:25:38.994 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2024-09-17T23:25:38.994 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:25:39.047 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:25:39.047 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:25:39.048 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:25:39.048 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:25:39.048 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:25:39.049 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:25:39.049 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 94 = '' 2024-09-17T23:25:39.049 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:25:39.049 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1.6 2024-09-17T23:25:40.650 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_5.status 2024-09-17T23:25:40.651 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:25:40.651 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:25:40.651 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_5 2024-09-17T23:25:40.651 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_5.status 2024-09-17T23:25:40.651 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2024-09-17T23:25:40.651 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:25:40.703 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:25:40.706 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id=94 2024-09-17T23:25:40.706 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 94 = 94 2024-09-17T23:25:40.706 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:25:40.706 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snap_present cluster1 mirror image_5 snap 2024-09-17T23:25:40.706 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:25:40.706 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:25:40.706 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_5 2024-09-17T23:25:40.706 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:25:40.706 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:25:40.706 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-17T23:25:40.706 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:25:41.707 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 info mirror/image_5@snap 2024-09-17T23:25:41.755 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'image_5': 2024-09-17T23:25:41.755 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 128 MiB in 32 objects 2024-09-17T23:25:41.755 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:25:41.755 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 2 2024-09-17T23:25:41.755 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 10c6dad6bcfe 2024-09-17T23:25:41.756 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.10c6dad6bcfe 2024-09-17T23:25:41.756 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:25:41.756 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-17T23:25:41.756 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:25:41.756 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:25:41.756 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:16:57 2024 2024-09-17T23:25:41.756 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:16:57 2024 2024-09-17T23:25:41.756 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:16:57 2024 2024-09-17T23:25:41.756 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: protected: False 2024-09-17T23:25:41.756 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:25:41.756 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:25:41.756 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: bde8fdf5-e3d2-42d5-b63a-37026e062eaa 2024-09-17T23:25:41.756 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: false 2024-09-17T23:25:41.759 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:25:41.759 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ compare_image_snaps mirror image_5 snap 2024-09-17T23:25:41.759 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:25:41.759 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_5 2024-09-17T23:25:41.759 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:25:41.759 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local ret=0 2024-09-17T23:25:41.759 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local rmt_export=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_5.export 2024-09-17T23:25:41.759 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local loc_export=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_5.export 2024-09-17T23:25:41.759 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_5.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_5.export 2024-09-17T23:25:41.760 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror export image_5@snap /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_5.export 2024-09-17T23:25:41.934 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-17T23:25:41.942 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 -p mirror export image_5@snap /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_5.export 2024-09-17T23:25:42.144 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-17T23:25:42.150 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ cmp /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_5.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_5.export 2024-09-17T23:25:42.213 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_5.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_5.export 2024-09-17T23:25:42.244 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:25:42.244 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:25:42.244 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_6 2024-09-17T23:25:42.244 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_snap cluster2 mirror image_6 snap 2024-09-17T23:25:42.245 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:25:42.245 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:25:42.245 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_6 2024-09-17T23:25:42.245 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:25:42.245 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror snap create image_6@snap --debug-rbd=20 --debug-journaler=20 2024-09-17T23:25:43.946 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_started cluster1 mirror image_6 2024-09-17T23:25:43.946 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:25:43.947 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:25:43.947 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_6 2024-09-17T23:25:43.947 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_state cluster1 mirror image_6 started 2024-09-17T23:25:43.947 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:25:43.947 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:25:43.947 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_6 2024-09-17T23:25:43.947 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=started 2024-09-17T23:25:43.947 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:25:43.947 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:25:43.947 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:25:44.947 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_replay_state cluster1 mirror image_6 started 2024-09-17T23:25:44.948 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:25:44.948 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:25:44.948 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_6 2024-09-17T23:25:44.948 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=started 2024-09-17T23:25:44.948 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_result 2024-09-17T23:25:44.948 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=stopped 2024-09-17T23:25:44.949 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_6 2024-09-17T23:25:44.950 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_instance=cluster1 2024-09-17T23:25:44.950 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:25:44.950 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:25:44.950 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ grep -i state 2024-09-17T23:25:44.950 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance=cluster1 2024-09-17T23:25:44.950 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local loop_instance 2024-09-17T23:25:44.950 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:25:44.950 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sleep 0 2024-09-17T23:25:44.950 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-17T23:25:44.950 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ seq 0 3 2024-09-17T23:25:44.952 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:25:44.952 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_6 2024-09-17T23:25:44.952 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:0 2024-09-17T23:25:44.952 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:25:44.952 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:25:44.952 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-17T23:25:44.952 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:0 2024-09-17T23:25:44.952 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:25:44.952 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:25:44.952 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:25:44.953 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=0 2024-09-17T23:25:44.953 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-17T23:25:44.953 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:25:44.953 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:25:44.953 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=0 2024-09-17T23:25:44.953 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:25:44.953 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-17T23:25:44.953 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:0 2024-09-17T23:25:44.953 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:25:44.953 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:25:44.953 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-17T23:25:44.953 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:0 2024-09-17T23:25:44.953 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:25:44.953 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:25:44.954 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:25:44.954 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:25:44.954 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-17T23:25:44.955 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:25:44.955 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:25:44.955 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=0 2024-09-17T23:25:44.955 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=0 2024-09-17T23:25:44.955 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-17T23:25:44.972 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:25:44.973 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:25:44.973 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:25:44.973 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_6 2024-09-17T23:25:45.098 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ return 0 2024-09-17T23:25:45.099 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ status_result=' "state": "Replaying"' 2024-09-17T23:25:45.099 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ echo ' "state": "Replaying"' 2024-09-17T23:25:45.099 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep -i Replaying 2024-09-17T23:25:45.100 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: "state": "Replaying" 2024-09-17T23:25:45.101 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_state=started 2024-09-17T23:25:45.101 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test started = started 2024-09-17T23:25:45.102 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:25:45.102 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_6 2024-09-17T23:25:45.102 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:25:45.102 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:25:45.102 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:25:45.102 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_6 2024-09-17T23:25:45.102 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = journal ']' 2024-09-17T23:25:45.102 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = snapshot ']' 2024-09-17T23:25:45.102 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_6 2024-09-17T23:25:45.102 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:25:45.102 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:25:45.102 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:25:45.102 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_6 2024-09-17T23:25:45.102 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster2-mirror-image_6.status 2024-09-17T23:25:45.103 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster2-mirror-image_6.status 2024-09-17T23:25:45.103 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:25:45.103 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_6.status 2024-09-17T23:25:45.103 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster1-mirror-image_6.status 2024-09-17T23:25:45.103 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster1-mirror-image_6.status 2024-09-17T23:25:45.104 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:25:45.104 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_status_log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_6.status 2024-09-17T23:25:45.104 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ mirror_image_snapshot cluster2 mirror image_6 2024-09-17T23:25:45.104 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:25:45.105 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:25:45.105 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_6 2024-09-17T23:25:45.105 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_6 2024-09-17T23:25:45.947 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Snapshot ID: 96 2024-09-17T23:25:45.959 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_6 /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_6.status 2024-09-17T23:25:45.959 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:25:45.959 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:25:45.959 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_6 2024-09-17T23:25:45.959 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_6.status 2024-09-17T23:25:45.959 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_6 --format xml 2024-09-17T23:25:45.959 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:25:46.005 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-17T23:25:46.007 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snapshot_id=96 2024-09-17T23:25:46.007 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:25:46.007 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:25:46.007 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.2 2024-09-17T23:25:46.208 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_6.status 2024-09-17T23:25:46.209 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:25:46.209 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:25:46.209 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_6 2024-09-17T23:25:46.209 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_6.status 2024-09-17T23:25:46.209 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2024-09-17T23:25:46.209 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:25:46.258 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:25:46.258 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:25:46.259 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:25:46.259 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:25:46.259 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:25:46.260 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:25:46.260 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 96 = '' 2024-09-17T23:25:46.260 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:25:46.260 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.4 2024-09-17T23:25:46.661 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_6.status 2024-09-17T23:25:46.661 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:25:46.661 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:25:46.661 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_6 2024-09-17T23:25:46.662 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_6.status 2024-09-17T23:25:46.662 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2024-09-17T23:25:46.662 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:25:46.710 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:25:46.711 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:25:46.713 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:25:46.713 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:25:46.713 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:25:46.713 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:25:46.713 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 96 = '' 2024-09-17T23:25:46.713 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:25:46.713 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.8 2024-09-17T23:25:47.514 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_6.status 2024-09-17T23:25:47.514 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:25:47.515 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:25:47.515 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_6 2024-09-17T23:25:47.515 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_6.status 2024-09-17T23:25:47.515 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2024-09-17T23:25:47.515 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:25:47.560 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:25:47.562 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id=96 2024-09-17T23:25:47.562 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 96 = 96 2024-09-17T23:25:47.562 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:25:47.563 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snap_present cluster1 mirror image_6 snap 2024-09-17T23:25:47.563 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:25:47.563 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:25:47.563 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_6 2024-09-17T23:25:47.563 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:25:47.563 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:25:47.563 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-17T23:25:47.563 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:25:48.563 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 info mirror/image_6@snap 2024-09-17T23:25:48.610 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'image_6': 2024-09-17T23:25:48.611 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 128 MiB in 32 objects 2024-09-17T23:25:48.611 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:25:48.611 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 2 2024-09-17T23:25:48.611 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 10c612199565 2024-09-17T23:25:48.611 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.10c612199565 2024-09-17T23:25:48.611 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:25:48.611 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-17T23:25:48.611 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:25:48.611 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:25:48.611 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:17:07 2024 2024-09-17T23:25:48.611 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:17:07 2024 2024-09-17T23:25:48.611 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:17:07 2024 2024-09-17T23:25:48.612 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: protected: False 2024-09-17T23:25:48.612 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:25:48.612 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:25:48.612 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: d1b13a10-bcbb-4eb9-8b9a-30b9235cece2 2024-09-17T23:25:48.612 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: false 2024-09-17T23:25:48.614 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:25:48.615 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ compare_image_snaps mirror image_6 snap 2024-09-17T23:25:48.615 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:25:48.615 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_6 2024-09-17T23:25:48.615 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:25:48.615 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local ret=0 2024-09-17T23:25:48.615 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local rmt_export=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_6.export 2024-09-17T23:25:48.615 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local loc_export=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_6.export 2024-09-17T23:25:48.615 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_6.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_6.export 2024-09-17T23:25:48.615 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror export image_6@snap /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_6.export 2024-09-17T23:25:48.786 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-17T23:25:48.794 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 -p mirror export image_6@snap /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_6.export 2024-09-17T23:25:48.971 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-17T23:25:48.978 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ cmp /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_6.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_6.export 2024-09-17T23:25:49.040 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_6.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_6.export 2024-09-17T23:25:49.071 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:25:49.071 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:25:49.071 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_7 2024-09-17T23:25:49.072 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_snap cluster2 mirror image_7 snap 2024-09-17T23:25:49.072 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:25:49.072 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:25:49.072 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_7 2024-09-17T23:25:49.072 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:25:49.072 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror snap create image_7@snap --debug-rbd=20 --debug-journaler=20 2024-09-17T23:25:50.968 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_started cluster1 mirror image_7 2024-09-17T23:25:50.968 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:25:50.968 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:25:50.968 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_7 2024-09-17T23:25:50.968 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_state cluster1 mirror image_7 started 2024-09-17T23:25:50.968 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:25:50.968 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:25:50.968 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_7 2024-09-17T23:25:50.968 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=started 2024-09-17T23:25:50.968 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:25:50.968 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:25:50.968 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:25:51.969 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_replay_state cluster1 mirror image_7 started 2024-09-17T23:25:51.969 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:25:51.969 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:25:51.969 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_7 2024-09-17T23:25:51.969 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=started 2024-09-17T23:25:51.970 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_result 2024-09-17T23:25:51.970 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=stopped 2024-09-17T23:25:51.971 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_7 2024-09-17T23:25:51.971 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_instance=cluster1 2024-09-17T23:25:51.971 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ grep -i state 2024-09-17T23:25:51.971 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:25:51.971 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:25:51.971 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance=cluster1 2024-09-17T23:25:51.971 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local loop_instance 2024-09-17T23:25:51.971 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:25:51.971 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sleep 0 2024-09-17T23:25:51.971 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-17T23:25:51.973 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ seq 0 3 2024-09-17T23:25:51.973 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:25:51.973 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_7 2024-09-17T23:25:51.973 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:0 2024-09-17T23:25:51.973 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:25:51.973 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:25:51.973 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-17T23:25:51.973 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:0 2024-09-17T23:25:51.973 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:25:51.974 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:25:51.974 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:25:51.974 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=0 2024-09-17T23:25:51.974 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-17T23:25:51.974 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:25:51.974 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:25:51.974 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=0 2024-09-17T23:25:51.974 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:25:51.974 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-17T23:25:51.975 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:0 2024-09-17T23:25:51.975 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:25:51.976 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:25:51.976 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-17T23:25:51.976 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:0 2024-09-17T23:25:51.976 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:25:51.976 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:25:51.976 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:25:51.976 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:25:51.976 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-17T23:25:51.976 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:25:51.976 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:25:51.976 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=0 2024-09-17T23:25:51.976 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=0 2024-09-17T23:25:51.976 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-17T23:25:51.998 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:25:51.998 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:25:51.998 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:25:51.998 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_7 2024-09-17T23:25:52.127 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ return 0 2024-09-17T23:25:52.128 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ status_result=' "state": "Replaying"' 2024-09-17T23:25:52.128 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ echo ' "state": "Replaying"' 2024-09-17T23:25:52.128 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep -i Replaying 2024-09-17T23:25:52.129 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: "state": "Replaying" 2024-09-17T23:25:52.130 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_state=started 2024-09-17T23:25:52.130 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test started = started 2024-09-17T23:25:52.130 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:25:52.131 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_7 2024-09-17T23:25:52.131 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:25:52.131 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:25:52.131 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:25:52.131 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_7 2024-09-17T23:25:52.131 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = journal ']' 2024-09-17T23:25:52.131 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = snapshot ']' 2024-09-17T23:25:52.131 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_7 2024-09-17T23:25:52.131 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:25:52.131 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:25:52.131 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:25:52.131 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_7 2024-09-17T23:25:52.131 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster2-mirror-image_7.status 2024-09-17T23:25:52.131 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster2-mirror-image_7.status 2024-09-17T23:25:52.132 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:25:52.132 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_7.status 2024-09-17T23:25:52.132 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster1-mirror-image_7.status 2024-09-17T23:25:52.132 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster1-mirror-image_7.status 2024-09-17T23:25:52.132 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:25:52.133 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_status_log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_7.status 2024-09-17T23:25:52.133 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ mirror_image_snapshot cluster2 mirror image_7 2024-09-17T23:25:52.133 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:25:52.133 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:25:52.134 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_7 2024-09-17T23:25:52.134 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_7 2024-09-17T23:25:52.960 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Snapshot ID: 98 2024-09-17T23:25:52.973 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_7 /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_7.status 2024-09-17T23:25:52.973 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:25:52.973 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:25:52.973 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_7 2024-09-17T23:25:52.973 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_7.status 2024-09-17T23:25:52.973 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_7 --format xml 2024-09-17T23:25:52.973 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:25:53.019 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-17T23:25:53.021 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snapshot_id=98 2024-09-17T23:25:53.021 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:25:53.021 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:25:53.021 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.2 2024-09-17T23:25:53.222 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_7.status 2024-09-17T23:25:53.222 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:25:53.222 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:25:53.222 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_7 2024-09-17T23:25:53.222 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_7.status 2024-09-17T23:25:53.222 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2024-09-17T23:25:53.222 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:25:53.270 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:25:53.270 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:25:53.271 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:25:53.271 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:25:53.271 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:25:53.272 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:25:53.272 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 98 = '' 2024-09-17T23:25:53.272 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:25:53.272 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.4 2024-09-17T23:25:53.673 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_7.status 2024-09-17T23:25:53.674 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:25:53.674 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:25:53.674 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_7 2024-09-17T23:25:53.674 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_7.status 2024-09-17T23:25:53.674 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2024-09-17T23:25:53.674 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:25:53.726 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:25:53.726 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:25:53.728 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:25:53.728 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:25:53.728 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:25:53.728 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:25:53.729 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 98 = '' 2024-09-17T23:25:53.729 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:25:53.729 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.8 2024-09-17T23:25:54.530 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_7.status 2024-09-17T23:25:54.530 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:25:54.530 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:25:54.530 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_7 2024-09-17T23:25:54.530 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_7.status 2024-09-17T23:25:54.530 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2024-09-17T23:25:54.530 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:25:54.578 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:25:54.580 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id=98 2024-09-17T23:25:54.580 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 98 = 98 2024-09-17T23:25:54.580 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:25:54.580 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snap_present cluster1 mirror image_7 snap 2024-09-17T23:25:54.580 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:25:54.580 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:25:54.580 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_7 2024-09-17T23:25:54.580 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:25:54.580 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:25:54.580 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-17T23:25:54.581 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:25:55.581 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 info mirror/image_7@snap 2024-09-17T23:25:55.628 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'image_7': 2024-09-17T23:25:55.628 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 128 MiB in 32 objects 2024-09-17T23:25:55.628 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:25:55.628 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 2 2024-09-17T23:25:55.628 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 10c636df067a 2024-09-17T23:25:55.628 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.10c636df067a 2024-09-17T23:25:55.628 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:25:55.629 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-17T23:25:55.629 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:25:55.629 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:25:55.629 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:17:19 2024 2024-09-17T23:25:55.629 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:17:19 2024 2024-09-17T23:25:55.629 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:17:19 2024 2024-09-17T23:25:55.629 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: protected: False 2024-09-17T23:25:55.629 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:25:55.629 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:25:55.629 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: 3eb02a9d-e7e5-427f-9aed-20cdaea8a88e 2024-09-17T23:25:55.629 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: false 2024-09-17T23:25:55.632 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:25:55.632 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ compare_image_snaps mirror image_7 snap 2024-09-17T23:25:55.632 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:25:55.632 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_7 2024-09-17T23:25:55.633 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:25:55.633 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local ret=0 2024-09-17T23:25:55.633 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local rmt_export=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_7.export 2024-09-17T23:25:55.633 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local loc_export=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_7.export 2024-09-17T23:25:55.633 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_7.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_7.export 2024-09-17T23:25:55.633 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror export image_7@snap /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_7.export 2024-09-17T23:25:55.804 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-17T23:25:55.811 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 -p mirror export image_7@snap /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_7.export 2024-09-17T23:25:56.035 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-17T23:25:56.042 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ cmp /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_7.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_7.export 2024-09-17T23:25:56.110 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_7.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_7.export 2024-09-17T23:25:56.140 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:25:56.141 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:25:56.141 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_8 2024-09-17T23:25:56.141 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_snap cluster2 mirror image_8 snap 2024-09-17T23:25:56.141 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:25:56.141 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:25:56.141 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_8 2024-09-17T23:25:56.141 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:25:56.141 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror snap create image_8@snap --debug-rbd=20 --debug-journaler=20 2024-09-17T23:25:57.979 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_started cluster1 mirror image_8 2024-09-17T23:25:57.979 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:25:57.979 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:25:57.980 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_8 2024-09-17T23:25:57.980 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_state cluster1 mirror image_8 started 2024-09-17T23:25:57.980 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:25:57.980 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:25:57.980 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_8 2024-09-17T23:25:57.980 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=started 2024-09-17T23:25:57.980 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:25:57.980 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:25:57.980 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:25:58.981 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_replay_state cluster1 mirror image_8 started 2024-09-17T23:25:58.981 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:25:58.981 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:25:58.981 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_8 2024-09-17T23:25:58.981 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=started 2024-09-17T23:25:58.981 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_result 2024-09-17T23:25:58.981 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=stopped 2024-09-17T23:25:58.982 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_8 2024-09-17T23:25:58.983 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_instance=cluster1 2024-09-17T23:25:58.983 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:25:58.983 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ grep -i state 2024-09-17T23:25:58.983 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:25:58.983 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance=cluster1 2024-09-17T23:25:58.983 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local loop_instance 2024-09-17T23:25:58.983 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:25:58.983 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sleep 0 2024-09-17T23:25:58.984 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-17T23:25:58.984 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ seq 0 3 2024-09-17T23:25:58.987 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:25:58.987 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_8 2024-09-17T23:25:58.987 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:0 2024-09-17T23:25:58.987 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:25:58.987 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:25:58.987 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-17T23:25:58.988 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:0 2024-09-17T23:25:58.988 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:25:58.988 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:25:58.988 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:25:58.988 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=0 2024-09-17T23:25:58.988 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-17T23:25:58.988 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:25:58.988 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:25:58.989 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=0 2024-09-17T23:25:58.989 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:25:58.989 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-17T23:25:58.989 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:0 2024-09-17T23:25:58.989 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:25:58.989 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:25:58.989 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-17T23:25:58.989 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:0 2024-09-17T23:25:58.989 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:25:58.989 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:25:58.989 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:25:58.990 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:25:58.990 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-17T23:25:58.990 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:25:58.990 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:25:58.991 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=0 2024-09-17T23:25:58.991 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=0 2024-09-17T23:25:58.991 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-17T23:25:59.007 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:25:59.007 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:25:59.007 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:25:59.007 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_8 2024-09-17T23:25:59.134 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ return 0 2024-09-17T23:25:59.135 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ status_result=' "state": "Replaying"' 2024-09-17T23:25:59.135 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ echo ' "state": "Replaying"' 2024-09-17T23:25:59.135 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep -i Replaying 2024-09-17T23:25:59.136 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: "state": "Replaying" 2024-09-17T23:25:59.137 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_state=started 2024-09-17T23:25:59.138 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test started = started 2024-09-17T23:25:59.138 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:25:59.138 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_8 2024-09-17T23:25:59.138 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:25:59.138 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:25:59.138 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:25:59.138 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_8 2024-09-17T23:25:59.138 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = journal ']' 2024-09-17T23:25:59.138 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = snapshot ']' 2024-09-17T23:25:59.138 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_8 2024-09-17T23:25:59.138 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:25:59.138 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:25:59.138 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:25:59.139 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_8 2024-09-17T23:25:59.139 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster2-mirror-image_8.status 2024-09-17T23:25:59.139 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster2-mirror-image_8.status 2024-09-17T23:25:59.139 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:25:59.139 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_8.status 2024-09-17T23:25:59.139 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster1-mirror-image_8.status 2024-09-17T23:25:59.140 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster1-mirror-image_8.status 2024-09-17T23:25:59.140 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:25:59.140 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_status_log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_8.status 2024-09-17T23:25:59.140 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ mirror_image_snapshot cluster2 mirror image_8 2024-09-17T23:25:59.141 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:25:59.141 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:25:59.141 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_8 2024-09-17T23:25:59.141 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_8 2024-09-17T23:25:59.995 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Snapshot ID: 100 2024-09-17T23:26:00.008 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_8 /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_8.status 2024-09-17T23:26:00.009 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:26:00.009 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:26:00.009 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_8 2024-09-17T23:26:00.009 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_8.status 2024-09-17T23:26:00.009 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_8 --format xml 2024-09-17T23:26:00.009 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:26:00.059 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-17T23:26:00.061 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snapshot_id=100 2024-09-17T23:26:00.061 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:26:00.061 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:26:00.061 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.2 2024-09-17T23:26:00.262 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_8.status 2024-09-17T23:26:00.263 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:26:00.263 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:26:00.263 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_8 2024-09-17T23:26:00.263 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_8.status 2024-09-17T23:26:00.263 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2024-09-17T23:26:00.263 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:26:00.312 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:26:00.312 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:26:00.313 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:26:00.313 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:26:00.313 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:26:00.314 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:26:00.314 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 100 = '' 2024-09-17T23:26:00.314 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:26:00.314 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.4 2024-09-17T23:26:00.715 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_8.status 2024-09-17T23:26:00.715 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:26:00.716 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:26:00.716 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_8 2024-09-17T23:26:00.716 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_8.status 2024-09-17T23:26:00.716 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2024-09-17T23:26:00.716 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:26:00.769 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:26:00.770 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:26:00.771 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:26:00.771 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:26:00.771 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:26:00.772 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:26:00.772 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 100 = '' 2024-09-17T23:26:00.772 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:26:00.772 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.8 2024-09-17T23:26:01.573 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_8.status 2024-09-17T23:26:01.573 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:26:01.573 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:26:01.573 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_8 2024-09-17T23:26:01.573 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_8.status 2024-09-17T23:26:01.574 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2024-09-17T23:26:01.574 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:26:01.622 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:26:01.624 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id=100 2024-09-17T23:26:01.624 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 100 = 100 2024-09-17T23:26:01.624 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:26:01.624 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snap_present cluster1 mirror image_8 snap 2024-09-17T23:26:01.624 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:26:01.624 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:26:01.624 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_8 2024-09-17T23:26:01.624 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:26:01.624 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:26:01.624 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-17T23:26:01.625 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:26:02.625 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 info mirror/image_8@snap 2024-09-17T23:26:02.673 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'image_8': 2024-09-17T23:26:02.673 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 128 MiB in 32 objects 2024-09-17T23:26:02.673 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:26:02.673 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 2 2024-09-17T23:26:02.673 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 10c6a6a12af2 2024-09-17T23:26:02.674 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.10c6a6a12af2 2024-09-17T23:26:02.674 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:26:02.674 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-17T23:26:02.674 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:26:02.674 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:26:02.674 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:17:27 2024 2024-09-17T23:26:02.674 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:17:27 2024 2024-09-17T23:26:02.674 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:17:27 2024 2024-09-17T23:26:02.674 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: protected: False 2024-09-17T23:26:02.674 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:26:02.674 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:26:02.674 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: ffca36ca-8ea2-4daa-91fd-451fc0c48309 2024-09-17T23:26:02.674 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: false 2024-09-17T23:26:02.677 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:26:02.677 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ compare_image_snaps mirror image_8 snap 2024-09-17T23:26:02.678 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:26:02.678 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_8 2024-09-17T23:26:02.678 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:26:02.678 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local ret=0 2024-09-17T23:26:02.678 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local rmt_export=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_8.export 2024-09-17T23:26:02.678 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local loc_export=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_8.export 2024-09-17T23:26:02.678 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_8.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_8.export 2024-09-17T23:26:02.678 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror export image_8@snap /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_8.export 2024-09-17T23:26:02.842 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-17T23:26:02.849 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 -p mirror export image_8@snap /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_8.export 2024-09-17T23:26:03.078 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-17T23:26:03.084 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ cmp /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_8.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_8.export 2024-09-17T23:26:03.146 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_8.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_8.export 2024-09-17T23:26:03.178 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:26:03.178 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:26:03.178 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_9 2024-09-17T23:26:03.178 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_snap cluster2 mirror image_9 snap 2024-09-17T23:26:03.178 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:26:03.178 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:26:03.178 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_9 2024-09-17T23:26:03.178 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:26:03.178 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror snap create image_9@snap --debug-rbd=20 --debug-journaler=20 2024-09-17T23:26:05.046 INFO:tasks.ceph.cluster2.osd.0.smithi038.stderr:2024-09-17T23:26:05.039+0000 7fa26a72e640 -1 reset not still connected to 0x55b3121c6820 2024-09-17T23:26:05.047 INFO:tasks.ceph.cluster2.osd.0.smithi038.stderr:2024-09-17T23:26:05.044+0000 7fa26a72e640 -1 reset not still connected to 0x55b31910fc70 2024-09-17T23:26:05.048 INFO:tasks.ceph.cluster2.osd.0.smithi038.stderr:2024-09-17T23:26:05.044+0000 7fa26a72e640 -1 reset not still connected to 0x55b31b652750 2024-09-17T23:26:05.048 INFO:tasks.ceph.cluster2.osd.0.smithi038.stderr:2024-09-17T23:26:05.044+0000 7fa26a72e640 -1 reset not still connected to 0x55b31c0fcd00 2024-09-17T23:26:05.048 INFO:tasks.ceph.cluster2.osd.0.smithi038.stderr:2024-09-17T23:26:05.044+0000 7fa26a72e640 -1 reset not still connected to 0x55b31c0fd6c0 2024-09-17T23:26:05.048 INFO:tasks.ceph.cluster2.osd.0.smithi038.stderr:2024-09-17T23:26:05.044+0000 7fa26a72e640 -1 reset not still connected to 0x55b31c0fdee0 2024-09-17T23:26:05.048 INFO:tasks.ceph.cluster2.osd.0.smithi038.stderr:2024-09-17T23:26:05.044+0000 7fa26a72e640 -1 reset not still connected to 0x55b323997c70 2024-09-17T23:26:05.048 INFO:tasks.ceph.cluster2.osd.0.smithi038.stderr:2024-09-17T23:26:05.044+0000 7fa26a72e640 -1 reset not still connected to 0x55b323e6ac30 2024-09-17T23:26:05.051 INFO:tasks.ceph.cluster2.osd.0.smithi038.stderr:2024-09-17T23:26:05.048+0000 7fa26a72e640 -1 reset not still connected to 0x55b326a52820 2024-09-17T23:26:05.051 INFO:tasks.ceph.cluster2.osd.0.smithi038.stderr:2024-09-17T23:26:05.048+0000 7fa26a72e640 -1 reset not still connected to 0x55b326a65c70 2024-09-17T23:26:05.051 INFO:tasks.ceph.cluster2.osd.0.smithi038.stderr:2024-09-17T23:26:05.048+0000 7fa26a72e640 -1 reset not still connected to 0x55b326c11930 2024-09-17T23:26:05.055 INFO:tasks.ceph.cluster2.osd.0.smithi038.stderr:2024-09-17T23:26:05.053+0000 7fa26a72e640 -1 reset not still connected to 0x55b328baa4e0 2024-09-17T23:26:05.055 INFO:tasks.ceph.cluster2.osd.0.smithi038.stderr:2024-09-17T23:26:05.053+0000 7fa26a72e640 -1 reset not still connected to 0x55b329649c70 2024-09-17T23:26:05.055 INFO:tasks.ceph.cluster2.osd.0.smithi038.stderr:2024-09-17T23:26:05.053+0000 7fa26a72e640 -1 reset not still connected to 0x55b32ddeef70 2024-09-17T23:26:05.055 INFO:tasks.ceph.cluster2.osd.0.smithi038.stderr:2024-09-17T23:26:05.053+0000 7fa26a72e640 -1 reset not still connected to 0x55b32e1005b0 2024-09-17T23:26:05.056 INFO:tasks.ceph.cluster2.osd.0.smithi038.stderr:2024-09-17T23:26:05.053+0000 7fa26a72e640 -1 reset not still connected to 0x55b330241a00 2024-09-17T23:26:05.056 INFO:tasks.ceph.cluster2.osd.0.smithi038.stderr:2024-09-17T23:26:05.053+0000 7fa26a72e640 -1 reset not still connected to 0x55b330e76680 2024-09-17T23:26:05.056 INFO:tasks.ceph.cluster2.osd.0.smithi038.stderr:2024-09-17T23:26:05.053+0000 7fa26a72e640 -1 reset not still connected to 0x55b330e95c70 2024-09-17T23:26:05.056 INFO:tasks.ceph.cluster2.osd.0.smithi038.stderr:2024-09-17T23:26:05.053+0000 7fa26a72e640 -1 reset not still connected to 0x55b3315b3520 2024-09-17T23:26:05.064 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_started cluster1 mirror image_9 2024-09-17T23:26:05.064 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:26:05.064 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:26:05.064 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_9 2024-09-17T23:26:05.064 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_state cluster1 mirror image_9 started 2024-09-17T23:26:05.064 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:26:05.065 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:26:05.065 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_9 2024-09-17T23:26:05.065 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=started 2024-09-17T23:26:05.065 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:26:05.065 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:26:05.065 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:26:06.065 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_replay_state cluster1 mirror image_9 started 2024-09-17T23:26:06.066 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:26:06.066 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:26:06.066 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_9 2024-09-17T23:26:06.066 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=started 2024-09-17T23:26:06.066 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_result 2024-09-17T23:26:06.066 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=stopped 2024-09-17T23:26:06.067 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_9 2024-09-17T23:26:06.067 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_instance=cluster1 2024-09-17T23:26:06.067 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ grep -i state 2024-09-17T23:26:06.067 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:26:06.067 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:26:06.067 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance=cluster1 2024-09-17T23:26:06.068 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local loop_instance 2024-09-17T23:26:06.068 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:26:06.068 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sleep 0 2024-09-17T23:26:06.068 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-17T23:26:06.070 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ seq 0 3 2024-09-17T23:26:06.070 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:26:06.070 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_9 2024-09-17T23:26:06.070 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:0 2024-09-17T23:26:06.070 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:26:06.070 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:26:06.070 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-17T23:26:06.071 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:0 2024-09-17T23:26:06.071 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:26:06.071 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:26:06.071 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:26:06.071 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=0 2024-09-17T23:26:06.071 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-17T23:26:06.071 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:26:06.071 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:26:06.071 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=0 2024-09-17T23:26:06.071 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:26:06.071 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-17T23:26:06.071 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:0 2024-09-17T23:26:06.071 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:26:06.071 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:26:06.071 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-17T23:26:06.071 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:0 2024-09-17T23:26:06.071 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:26:06.072 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:26:06.072 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:26:06.072 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:26:06.072 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-17T23:26:06.072 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:26:06.072 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:26:06.072 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=0 2024-09-17T23:26:06.072 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=0 2024-09-17T23:26:06.072 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-17T23:26:06.092 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:26:06.092 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:26:06.093 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:26:06.093 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_9 2024-09-17T23:26:06.220 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ return 0 2024-09-17T23:26:06.221 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ status_result=' "state": "Replaying"' 2024-09-17T23:26:06.221 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ echo ' "state": "Replaying"' 2024-09-17T23:26:06.221 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep -i Replaying 2024-09-17T23:26:06.222 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: "state": "Replaying" 2024-09-17T23:26:06.224 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_state=started 2024-09-17T23:26:06.224 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test started = started 2024-09-17T23:26:06.224 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:26:06.224 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_9 2024-09-17T23:26:06.225 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:26:06.225 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:26:06.225 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:26:06.225 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_9 2024-09-17T23:26:06.225 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = journal ']' 2024-09-17T23:26:06.225 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = snapshot ']' 2024-09-17T23:26:06.225 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_9 2024-09-17T23:26:06.225 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:26:06.225 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:26:06.225 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:26:06.225 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_9 2024-09-17T23:26:06.225 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster2-mirror-image_9.status 2024-09-17T23:26:06.226 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster2-mirror-image_9.status 2024-09-17T23:26:06.226 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:26:06.226 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_9.status 2024-09-17T23:26:06.226 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster1-mirror-image_9.status 2024-09-17T23:26:06.227 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster1-mirror-image_9.status 2024-09-17T23:26:06.227 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:26:06.227 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_status_log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_9.status 2024-09-17T23:26:06.227 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ mirror_image_snapshot cluster2 mirror image_9 2024-09-17T23:26:06.227 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:26:06.227 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:26:06.227 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_9 2024-09-17T23:26:06.227 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_9 2024-09-17T23:26:07.045 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Snapshot ID: 102 2024-09-17T23:26:07.057 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_9 /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_9.status 2024-09-17T23:26:07.058 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:26:07.058 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:26:07.058 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_9 2024-09-17T23:26:07.058 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_9.status 2024-09-17T23:26:07.058 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_9 --format xml 2024-09-17T23:26:07.058 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:26:07.111 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-17T23:26:07.113 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snapshot_id=102 2024-09-17T23:26:07.113 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:26:07.113 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:26:07.113 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.2 2024-09-17T23:26:07.314 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_9.status 2024-09-17T23:26:07.315 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:26:07.315 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:26:07.315 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_9 2024-09-17T23:26:07.315 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_9.status 2024-09-17T23:26:07.315 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2024-09-17T23:26:07.315 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:26:07.363 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:26:07.363 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:26:07.365 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:26:07.365 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:26:07.365 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:26:07.365 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:26:07.366 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 102 = '' 2024-09-17T23:26:07.366 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:26:07.366 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.4 2024-09-17T23:26:07.767 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_9.status 2024-09-17T23:26:07.767 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:26:07.767 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:26:07.767 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_9 2024-09-17T23:26:07.767 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_9.status 2024-09-17T23:26:07.767 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2024-09-17T23:26:07.767 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:26:07.819 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:26:07.819 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:26:07.820 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:26:07.821 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:26:07.821 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:26:07.821 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:26:07.821 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 102 = '' 2024-09-17T23:26:07.822 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:26:07.822 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.8 2024-09-17T23:26:08.623 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_9.status 2024-09-17T23:26:08.623 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:26:08.623 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:26:08.623 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_9 2024-09-17T23:26:08.623 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_9.status 2024-09-17T23:26:08.623 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2024-09-17T23:26:08.623 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:26:08.671 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:26:08.673 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id=102 2024-09-17T23:26:08.673 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 102 = 102 2024-09-17T23:26:08.673 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:26:08.673 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snap_present cluster1 mirror image_9 snap 2024-09-17T23:26:08.673 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:26:08.673 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:26:08.674 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_9 2024-09-17T23:26:08.674 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:26:08.674 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:26:08.674 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-17T23:26:08.674 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:26:08.844 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.43786046106754095) > weight (0.25) 2024-09-17T23:26:08.844 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.5600277061721907) > weight (0.25) 2024-09-17T23:26:08.844 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.5154885512782208) > weight (0.25) 2024-09-17T23:26:08.844 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.46997239422012105) > weight (0.25) 2024-09-17T23:26:08.844 INFO:tasks.rbd_mirror_thrash:waiting for 104 secs before thrashing 2024-09-17T23:26:09.674 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 info mirror/image_9@snap 2024-09-17T23:26:09.728 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'image_9': 2024-09-17T23:26:09.728 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 128 MiB in 32 objects 2024-09-17T23:26:09.728 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:26:09.728 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 2 2024-09-17T23:26:09.728 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 10c67c64a3b0 2024-09-17T23:26:09.729 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.10c67c64a3b0 2024-09-17T23:26:09.729 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:26:09.729 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-17T23:26:09.729 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:26:09.729 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:26:09.729 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:17:34 2024 2024-09-17T23:26:09.729 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:17:34 2024 2024-09-17T23:26:09.729 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:17:34 2024 2024-09-17T23:26:09.729 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: protected: False 2024-09-17T23:26:09.729 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:26:09.729 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:26:09.729 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: bdde2c0b-4959-43dc-9438-c5f0c88af81b 2024-09-17T23:26:09.729 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: false 2024-09-17T23:26:09.732 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:26:09.732 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ compare_image_snaps mirror image_9 snap 2024-09-17T23:26:09.732 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:26:09.733 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_9 2024-09-17T23:26:09.733 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:26:09.733 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local ret=0 2024-09-17T23:26:09.733 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local rmt_export=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_9.export 2024-09-17T23:26:09.733 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local loc_export=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_9.export 2024-09-17T23:26:09.733 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_9.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_9.export 2024-09-17T23:26:09.733 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror export image_9@snap /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_9.export 2024-09-17T23:26:09.911 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-17T23:26:09.916 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 -p mirror export image_9@snap /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_9.export 2024-09-17T23:26:10.139 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-17T23:26:10.147 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ cmp /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_9.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_9.export 2024-09-17T23:26:10.209 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_9.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_9.export 2024-09-17T23:26:10.240 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:26:10.241 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:26:10.241 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_10 2024-09-17T23:26:10.241 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_snap cluster2 mirror image_10 snap 2024-09-17T23:26:10.241 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:26:10.241 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:26:10.241 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_10 2024-09-17T23:26:10.241 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:26:10.241 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror snap create image_10@snap --debug-rbd=20 --debug-journaler=20 2024-09-17T23:26:12.073 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_started cluster1 mirror image_10 2024-09-17T23:26:12.074 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:26:12.074 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:26:12.074 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_10 2024-09-17T23:26:12.074 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_state cluster1 mirror image_10 started 2024-09-17T23:26:12.074 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:26:12.074 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:26:12.074 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_10 2024-09-17T23:26:12.074 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=started 2024-09-17T23:26:12.074 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:26:12.074 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:26:12.074 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:26:13.074 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_replay_state cluster1 mirror image_10 started 2024-09-17T23:26:13.075 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:26:13.075 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:26:13.075 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_10 2024-09-17T23:26:13.075 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=started 2024-09-17T23:26:13.075 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_result 2024-09-17T23:26:13.075 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=stopped 2024-09-17T23:26:13.076 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_10 2024-09-17T23:26:13.076 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ grep -i state 2024-09-17T23:26:13.076 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_instance=cluster1 2024-09-17T23:26:13.076 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:26:13.076 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:26:13.077 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance=cluster1 2024-09-17T23:26:13.077 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local loop_instance 2024-09-17T23:26:13.077 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:26:13.077 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sleep 0 2024-09-17T23:26:13.077 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-17T23:26:13.078 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ seq 0 3 2024-09-17T23:26:13.078 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:26:13.078 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_10 2024-09-17T23:26:13.079 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:0 2024-09-17T23:26:13.079 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:26:13.079 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:26:13.079 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-17T23:26:13.079 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:0 2024-09-17T23:26:13.079 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:26:13.079 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:26:13.079 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:26:13.079 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=0 2024-09-17T23:26:13.079 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-17T23:26:13.079 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:26:13.079 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:26:13.079 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=0 2024-09-17T23:26:13.079 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:26:13.080 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-17T23:26:13.081 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:0 2024-09-17T23:26:13.081 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:26:13.081 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:26:13.081 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-17T23:26:13.081 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:0 2024-09-17T23:26:13.081 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:26:13.081 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:26:13.081 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:26:13.082 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:26:13.082 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-17T23:26:13.082 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:26:13.082 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:26:13.082 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=0 2024-09-17T23:26:13.082 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=0 2024-09-17T23:26:13.082 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-17T23:26:13.102 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:26:13.102 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:26:13.102 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:26:13.102 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_10 2024-09-17T23:26:13.226 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ return 0 2024-09-17T23:26:13.227 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ status_result=' "state": "Replaying"' 2024-09-17T23:26:13.227 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ echo ' "state": "Replaying"' 2024-09-17T23:26:13.227 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep -i Replaying 2024-09-17T23:26:13.228 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: "state": "Replaying" 2024-09-17T23:26:13.229 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_state=started 2024-09-17T23:26:13.229 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test started = started 2024-09-17T23:26:13.229 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:26:13.229 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_10 2024-09-17T23:26:13.230 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:26:13.230 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:26:13.230 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:26:13.230 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_10 2024-09-17T23:26:13.230 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = journal ']' 2024-09-17T23:26:13.230 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = snapshot ']' 2024-09-17T23:26:13.230 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_10 2024-09-17T23:26:13.230 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:26:13.230 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:26:13.230 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:26:13.230 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_10 2024-09-17T23:26:13.231 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster2-mirror-image_10.status 2024-09-17T23:26:13.231 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster2-mirror-image_10.status 2024-09-17T23:26:13.231 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:26:13.232 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_10.status 2024-09-17T23:26:13.232 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster1-mirror-image_10.status 2024-09-17T23:26:13.232 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster1-mirror-image_10.status 2024-09-17T23:26:13.232 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:26:13.232 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_status_log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_10.status 2024-09-17T23:26:13.232 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ mirror_image_snapshot cluster2 mirror image_10 2024-09-17T23:26:13.232 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:26:13.233 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:26:13.233 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_10 2024-09-17T23:26:13.233 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_10 2024-09-17T23:26:14.079 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Snapshot ID: 104 2024-09-17T23:26:14.092 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_10 /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_10.status 2024-09-17T23:26:14.092 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:26:14.092 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:26:14.092 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_10 2024-09-17T23:26:14.092 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_10.status 2024-09-17T23:26:14.093 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_10 --format xml 2024-09-17T23:26:14.093 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:26:14.140 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-17T23:26:14.142 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snapshot_id=104 2024-09-17T23:26:14.142 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:26:14.142 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:26:14.142 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.2 2024-09-17T23:26:14.344 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_10.status 2024-09-17T23:26:14.344 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:26:14.344 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:26:14.344 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_10 2024-09-17T23:26:14.344 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_10.status 2024-09-17T23:26:14.344 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2024-09-17T23:26:14.344 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:26:14.395 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:26:14.395 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:26:14.397 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:26:14.397 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:26:14.397 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:26:14.397 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:26:14.397 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 104 = '' 2024-09-17T23:26:14.397 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:26:14.397 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.4 2024-09-17T23:26:14.799 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_10.status 2024-09-17T23:26:14.799 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:26:14.799 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:26:14.799 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_10 2024-09-17T23:26:14.799 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_10.status 2024-09-17T23:26:14.799 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2024-09-17T23:26:14.800 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:26:14.852 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:26:14.853 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:26:14.854 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:26:14.854 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:26:14.854 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:26:14.855 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:26:14.855 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 104 = '' 2024-09-17T23:26:14.855 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:26:14.855 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.8 2024-09-17T23:26:15.658 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_10.status 2024-09-17T23:26:15.658 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:26:15.658 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:26:15.658 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_10 2024-09-17T23:26:15.658 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_10.status 2024-09-17T23:26:15.658 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2024-09-17T23:26:15.658 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:26:15.703 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:26:15.706 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id=104 2024-09-17T23:26:15.706 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 104 = 104 2024-09-17T23:26:15.706 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:26:15.706 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snap_present cluster1 mirror image_10 snap 2024-09-17T23:26:15.706 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:26:15.706 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:26:15.706 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_10 2024-09-17T23:26:15.706 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:26:15.706 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:26:15.706 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-17T23:26:15.706 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:26:16.706 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 info mirror/image_10@snap 2024-09-17T23:26:16.756 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'image_10': 2024-09-17T23:26:16.756 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 128 MiB in 32 objects 2024-09-17T23:26:16.756 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:26:16.756 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 2 2024-09-17T23:26:16.756 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 10c6d8032b47 2024-09-17T23:26:16.756 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.10c6d8032b47 2024-09-17T23:26:16.756 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:26:16.756 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-17T23:26:16.756 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:26:16.756 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:26:16.756 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:17:49 2024 2024-09-17T23:26:16.757 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:17:49 2024 2024-09-17T23:26:16.757 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:17:49 2024 2024-09-17T23:26:16.757 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: protected: False 2024-09-17T23:26:16.757 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:26:16.757 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:26:16.757 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: 2dc39d68-70b6-4566-beb0-8904894734d5 2024-09-17T23:26:16.757 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: false 2024-09-17T23:26:16.760 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:26:16.760 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ compare_image_snaps mirror image_10 snap 2024-09-17T23:26:16.760 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:26:16.760 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_10 2024-09-17T23:26:16.760 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:26:16.760 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local ret=0 2024-09-17T23:26:16.760 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local rmt_export=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_10.export 2024-09-17T23:26:16.760 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local loc_export=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_10.export 2024-09-17T23:26:16.760 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_10.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_10.export 2024-09-17T23:26:16.761 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror export image_10@snap /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_10.export 2024-09-17T23:26:16.931 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-17T23:26:16.937 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 -p mirror export image_10@snap /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_10.export 2024-09-17T23:26:17.171 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-17T23:26:17.179 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ cmp /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_10.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_10.export 2024-09-17T23:26:17.241 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_10.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_10.export 2024-09-17T23:26:17.272 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:26:17.272 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:26:17.273 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_11 2024-09-17T23:26:17.273 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_snap cluster2 mirror image_11 snap 2024-09-17T23:26:17.273 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:26:17.273 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:26:17.273 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_11 2024-09-17T23:26:17.273 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:26:17.273 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror snap create image_11@snap --debug-rbd=20 --debug-journaler=20 2024-09-17T23:26:19.101 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_started cluster1 mirror image_11 2024-09-17T23:26:19.101 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:26:19.102 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:26:19.102 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_11 2024-09-17T23:26:19.102 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_state cluster1 mirror image_11 started 2024-09-17T23:26:19.102 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:26:19.102 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:26:19.102 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_11 2024-09-17T23:26:19.102 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=started 2024-09-17T23:26:19.102 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:26:19.102 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:26:19.102 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:26:20.103 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_replay_state cluster1 mirror image_11 started 2024-09-17T23:26:20.103 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:26:20.103 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:26:20.103 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_11 2024-09-17T23:26:20.103 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=started 2024-09-17T23:26:20.103 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_result 2024-09-17T23:26:20.103 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=stopped 2024-09-17T23:26:20.104 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_11 2024-09-17T23:26:20.104 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_instance=cluster1 2024-09-17T23:26:20.104 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:26:20.104 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:26:20.104 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance=cluster1 2024-09-17T23:26:20.104 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local loop_instance 2024-09-17T23:26:20.104 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ grep -i state 2024-09-17T23:26:20.105 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:26:20.105 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sleep 0 2024-09-17T23:26:20.105 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-17T23:26:20.105 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ seq 0 3 2024-09-17T23:26:20.108 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:26:20.108 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_11 2024-09-17T23:26:20.108 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:0 2024-09-17T23:26:20.108 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:26:20.108 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:26:20.108 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-17T23:26:20.108 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:0 2024-09-17T23:26:20.108 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:26:20.108 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:26:20.108 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:26:20.109 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=0 2024-09-17T23:26:20.109 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-17T23:26:20.109 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:26:20.109 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:26:20.109 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=0 2024-09-17T23:26:20.109 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:26:20.109 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-17T23:26:20.109 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:0 2024-09-17T23:26:20.109 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:26:20.109 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:26:20.109 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-17T23:26:20.109 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:0 2024-09-17T23:26:20.109 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:26:20.110 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:26:20.110 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:26:20.110 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:26:20.110 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-17T23:26:20.110 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:26:20.110 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:26:20.110 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=0 2024-09-17T23:26:20.110 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=0 2024-09-17T23:26:20.110 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-17T23:26:20.130 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:26:20.130 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:26:20.130 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:26:20.130 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_11 2024-09-17T23:26:20.256 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ return 0 2024-09-17T23:26:20.256 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ status_result=' "state": "Replaying"' 2024-09-17T23:26:20.257 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ echo ' "state": "Replaying"' 2024-09-17T23:26:20.257 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep -i Replaying 2024-09-17T23:26:20.257 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: "state": "Replaying" 2024-09-17T23:26:20.259 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_state=started 2024-09-17T23:26:20.259 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test started = started 2024-09-17T23:26:20.259 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:26:20.259 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_11 2024-09-17T23:26:20.259 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:26:20.259 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:26:20.259 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:26:20.259 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_11 2024-09-17T23:26:20.259 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = journal ']' 2024-09-17T23:26:20.259 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = snapshot ']' 2024-09-17T23:26:20.259 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_11 2024-09-17T23:26:20.260 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:26:20.260 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:26:20.260 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:26:20.260 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_11 2024-09-17T23:26:20.260 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster2-mirror-image_11.status 2024-09-17T23:26:20.260 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster2-mirror-image_11.status 2024-09-17T23:26:20.260 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:26:20.261 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_11.status 2024-09-17T23:26:20.261 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster1-mirror-image_11.status 2024-09-17T23:26:20.261 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster1-mirror-image_11.status 2024-09-17T23:26:20.261 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:26:20.261 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_status_log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_11.status 2024-09-17T23:26:20.262 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ mirror_image_snapshot cluster2 mirror image_11 2024-09-17T23:26:20.262 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:26:20.262 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:26:20.262 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_11 2024-09-17T23:26:20.262 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_11 2024-09-17T23:26:21.108 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Snapshot ID: 106 2024-09-17T23:26:21.121 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_11 /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_11.status 2024-09-17T23:26:21.121 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:26:21.121 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:26:21.121 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_11 2024-09-17T23:26:21.121 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_11.status 2024-09-17T23:26:21.121 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_11 --format xml 2024-09-17T23:26:21.121 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:26:21.172 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-17T23:26:21.172 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snapshot_id=106 2024-09-17T23:26:21.172 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:26:21.173 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:26:21.173 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.2 2024-09-17T23:26:21.374 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_11.status 2024-09-17T23:26:21.374 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:26:21.374 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:26:21.374 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_11 2024-09-17T23:26:21.374 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_11.status 2024-09-17T23:26:21.374 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2024-09-17T23:26:21.374 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:26:21.426 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:26:21.426 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:26:21.427 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:26:21.427 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:26:21.427 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:26:21.428 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:26:21.428 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 106 = '' 2024-09-17T23:26:21.428 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:26:21.428 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.4 2024-09-17T23:26:21.829 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_11.status 2024-09-17T23:26:21.830 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:26:21.830 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:26:21.830 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_11 2024-09-17T23:26:21.830 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_11.status 2024-09-17T23:26:21.830 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2024-09-17T23:26:21.830 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:26:21.883 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:26:21.883 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:26:21.885 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:26:21.885 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:26:21.885 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:26:21.885 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:26:21.885 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 106 = '' 2024-09-17T23:26:21.885 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:26:21.886 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.8 2024-09-17T23:26:22.686 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_11.status 2024-09-17T23:26:22.687 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:26:22.687 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:26:22.687 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_11 2024-09-17T23:26:22.687 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_11.status 2024-09-17T23:26:22.687 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2024-09-17T23:26:22.687 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:26:22.733 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:26:22.736 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id=106 2024-09-17T23:26:22.736 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 106 = 106 2024-09-17T23:26:22.736 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:26:22.736 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snap_present cluster1 mirror image_11 snap 2024-09-17T23:26:22.736 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:26:22.736 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:26:22.736 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_11 2024-09-17T23:26:22.736 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:26:22.736 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:26:22.736 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-17T23:26:22.737 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:26:23.737 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 info mirror/image_11@snap 2024-09-17T23:26:23.787 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'image_11': 2024-09-17T23:26:23.787 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 128 MiB in 32 objects 2024-09-17T23:26:23.787 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:26:23.788 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 2 2024-09-17T23:26:23.788 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 10c6d388fa12 2024-09-17T23:26:23.788 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.10c6d388fa12 2024-09-17T23:26:23.788 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:26:23.788 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-17T23:26:23.788 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:26:23.788 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:26:23.788 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:18:02 2024 2024-09-17T23:26:23.788 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:18:02 2024 2024-09-17T23:26:23.788 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:18:02 2024 2024-09-17T23:26:23.788 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: protected: False 2024-09-17T23:26:23.788 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:26:23.788 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:26:23.789 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: bd10d2f7-c8b2-4a32-94d9-b2af3d68bf2d 2024-09-17T23:26:23.789 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: false 2024-09-17T23:26:23.791 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:26:23.791 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ compare_image_snaps mirror image_11 snap 2024-09-17T23:26:23.791 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:26:23.791 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_11 2024-09-17T23:26:23.792 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:26:23.792 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local ret=0 2024-09-17T23:26:23.792 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local rmt_export=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_11.export 2024-09-17T23:26:23.792 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local loc_export=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_11.export 2024-09-17T23:26:23.792 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_11.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_11.export 2024-09-17T23:26:23.792 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror export image_11@snap /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_11.export 2024-09-17T23:26:24.164 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-17T23:26:24.171 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 -p mirror export image_11@snap /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_11.export 2024-09-17T23:26:24.392 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-17T23:26:24.398 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ cmp /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_11.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_11.export 2024-09-17T23:26:24.460 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_11.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_11.export 2024-09-17T23:26:24.491 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:26:24.491 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:26:24.491 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_12 2024-09-17T23:26:24.491 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_snap cluster2 mirror image_12 snap 2024-09-17T23:26:24.491 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:26:24.491 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:26:24.491 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_12 2024-09-17T23:26:24.491 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:26:24.492 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror snap create image_12@snap --debug-rbd=20 --debug-journaler=20 2024-09-17T23:26:26.120 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_started cluster1 mirror image_12 2024-09-17T23:26:26.120 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:26:26.120 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:26:26.120 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_12 2024-09-17T23:26:26.121 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_state cluster1 mirror image_12 started 2024-09-17T23:26:26.121 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:26:26.121 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:26:26.121 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_12 2024-09-17T23:26:26.121 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=started 2024-09-17T23:26:26.121 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:26:26.121 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:26:26.121 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:26:27.121 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_replay_state cluster1 mirror image_12 started 2024-09-17T23:26:27.121 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:26:27.121 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:26:27.122 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_12 2024-09-17T23:26:27.122 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=started 2024-09-17T23:26:27.122 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_result 2024-09-17T23:26:27.122 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=stopped 2024-09-17T23:26:27.123 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_12 2024-09-17T23:26:27.123 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_instance=cluster1 2024-09-17T23:26:27.123 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:26:27.123 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ grep -i state 2024-09-17T23:26:27.123 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:26:27.124 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance=cluster1 2024-09-17T23:26:27.124 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local loop_instance 2024-09-17T23:26:27.124 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:26:27.124 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sleep 0 2024-09-17T23:26:27.124 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-17T23:26:27.124 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ seq 0 3 2024-09-17T23:26:27.125 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:26:27.126 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_12 2024-09-17T23:26:27.126 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:0 2024-09-17T23:26:27.126 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:26:27.126 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:26:27.126 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-17T23:26:27.126 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:0 2024-09-17T23:26:27.126 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:26:27.126 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:26:27.126 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:26:27.126 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=0 2024-09-17T23:26:27.126 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-17T23:26:27.126 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:26:27.126 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:26:27.126 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=0 2024-09-17T23:26:27.127 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:26:27.127 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-17T23:26:27.128 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:0 2024-09-17T23:26:27.128 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:26:27.128 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:26:27.128 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-17T23:26:27.128 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:0 2024-09-17T23:26:27.128 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:26:27.128 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:26:27.128 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:26:27.128 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:26:27.129 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-17T23:26:27.129 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:26:27.129 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:26:27.129 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=0 2024-09-17T23:26:27.129 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=0 2024-09-17T23:26:27.129 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-17T23:26:27.147 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:26:27.147 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:26:27.147 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:26:27.147 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_12 2024-09-17T23:26:27.269 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ return 0 2024-09-17T23:26:27.270 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ status_result=' "state": "Replaying"' 2024-09-17T23:26:27.270 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ echo ' "state": "Replaying"' 2024-09-17T23:26:27.270 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep -i Replaying 2024-09-17T23:26:27.271 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: "state": "Replaying" 2024-09-17T23:26:27.272 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_state=started 2024-09-17T23:26:27.272 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test started = started 2024-09-17T23:26:27.272 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:26:27.273 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_12 2024-09-17T23:26:27.273 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:26:27.273 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:26:27.273 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:26:27.273 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_12 2024-09-17T23:26:27.273 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = journal ']' 2024-09-17T23:26:27.273 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = snapshot ']' 2024-09-17T23:26:27.273 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_12 2024-09-17T23:26:27.273 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:26:27.273 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:26:27.273 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:26:27.273 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_12 2024-09-17T23:26:27.273 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster2-mirror-image_12.status 2024-09-17T23:26:27.273 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster2-mirror-image_12.status 2024-09-17T23:26:27.274 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:26:27.274 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_12.status 2024-09-17T23:26:27.274 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster1-mirror-image_12.status 2024-09-17T23:26:27.274 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster1-mirror-image_12.status 2024-09-17T23:26:27.274 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:26:27.275 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_status_log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_12.status 2024-09-17T23:26:27.275 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ mirror_image_snapshot cluster2 mirror image_12 2024-09-17T23:26:27.275 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:26:27.275 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:26:27.276 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_12 2024-09-17T23:26:27.276 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_12 2024-09-17T23:26:28.139 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Snapshot ID: 108 2024-09-17T23:26:28.151 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_12 /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_12.status 2024-09-17T23:26:28.151 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:26:28.151 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:26:28.151 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_12 2024-09-17T23:26:28.152 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_12.status 2024-09-17T23:26:28.152 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_12 --format xml 2024-09-17T23:26:28.152 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:26:28.200 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-17T23:26:28.202 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snapshot_id=108 2024-09-17T23:26:28.202 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:26:28.203 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:26:28.203 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.2 2024-09-17T23:26:28.404 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_12.status 2024-09-17T23:26:28.404 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:26:28.404 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:26:28.404 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_12 2024-09-17T23:26:28.404 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_12.status 2024-09-17T23:26:28.404 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2024-09-17T23:26:28.405 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:26:28.455 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:26:28.456 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:26:28.457 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:26:28.457 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:26:28.457 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:26:28.457 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:26:28.458 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 108 = '' 2024-09-17T23:26:28.458 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:26:28.458 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.4 2024-09-17T23:26:28.859 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_12.status 2024-09-17T23:26:28.859 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:26:28.859 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:26:28.859 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_12 2024-09-17T23:26:28.859 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_12.status 2024-09-17T23:26:28.859 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2024-09-17T23:26:28.859 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:26:28.913 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:26:28.913 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:26:28.914 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:26:28.914 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:26:28.915 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:26:28.915 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:26:28.915 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 108 = '' 2024-09-17T23:26:28.915 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:26:28.915 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.8 2024-09-17T23:26:29.717 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_12.status 2024-09-17T23:26:29.717 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:26:29.717 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:26:29.717 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_12 2024-09-17T23:26:29.717 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_12.status 2024-09-17T23:26:29.717 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2024-09-17T23:26:29.717 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:26:29.767 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:26:29.769 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id=108 2024-09-17T23:26:29.769 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 108 = 108 2024-09-17T23:26:29.769 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:26:29.769 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snap_present cluster1 mirror image_12 snap 2024-09-17T23:26:29.769 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:26:29.769 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:26:29.769 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_12 2024-09-17T23:26:29.769 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:26:29.769 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:26:29.770 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-17T23:26:29.770 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:26:30.770 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 info mirror/image_12@snap 2024-09-17T23:26:30.820 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'image_12': 2024-09-17T23:26:30.820 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 128 MiB in 32 objects 2024-09-17T23:26:30.820 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:26:30.820 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 2 2024-09-17T23:26:30.820 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 10c634c2abad 2024-09-17T23:26:30.820 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.10c634c2abad 2024-09-17T23:26:30.820 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:26:30.820 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-17T23:26:30.820 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:26:30.820 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:26:30.820 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:18:12 2024 2024-09-17T23:26:30.821 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:18:12 2024 2024-09-17T23:26:30.821 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:18:12 2024 2024-09-17T23:26:30.821 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: protected: False 2024-09-17T23:26:30.821 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:26:30.821 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:26:30.821 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: 1e208d5a-c3ae-4b3c-bb32-593db1dbda33 2024-09-17T23:26:30.821 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: false 2024-09-17T23:26:30.823 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:26:30.823 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ compare_image_snaps mirror image_12 snap 2024-09-17T23:26:30.824 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:26:30.824 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_12 2024-09-17T23:26:30.824 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:26:30.824 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local ret=0 2024-09-17T23:26:30.824 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local rmt_export=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_12.export 2024-09-17T23:26:30.824 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local loc_export=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_12.export 2024-09-17T23:26:30.824 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_12.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_12.export 2024-09-17T23:26:30.824 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror export image_12@snap /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_12.export 2024-09-17T23:26:30.996 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-17T23:26:31.001 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 -p mirror export image_12@snap /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_12.export 2024-09-17T23:26:31.209 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-17T23:26:31.215 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ cmp /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_12.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_12.export 2024-09-17T23:26:31.278 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_12.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_12.export 2024-09-17T23:26:31.311 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:26:31.311 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:26:31.311 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_13 2024-09-17T23:26:31.311 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_snap cluster2 mirror image_13 snap 2024-09-17T23:26:31.311 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:26:31.311 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:26:31.311 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_13 2024-09-17T23:26:31.311 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:26:31.311 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror snap create image_13@snap --debug-rbd=20 --debug-journaler=20 2024-09-17T23:26:31.947 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.6393366983146688) > weight (0.25) 2024-09-17T23:26:31.947 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.9412454426559538) > weight (0.25) 2024-09-17T23:26:31.947 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2024-09-17T23:26:31.948 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2024-09-17T23:26:31.948 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2024-09-17T23:26:31.949 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi038.stderr:2024-09-17T23:26:31.947+0000 7f242ef50640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 47408) UID: 1000 2024-09-17T23:26:33.165 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_started cluster1 mirror image_13 2024-09-17T23:26:33.165 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:26:33.165 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:26:33.165 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_13 2024-09-17T23:26:33.165 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_state cluster1 mirror image_13 started 2024-09-17T23:26:33.165 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:26:33.165 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:26:33.165 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_13 2024-09-17T23:26:33.165 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=started 2024-09-17T23:26:33.165 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:26:33.165 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:26:33.166 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:26:34.166 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_replay_state cluster1 mirror image_13 started 2024-09-17T23:26:34.166 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:26:34.166 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:26:34.167 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_13 2024-09-17T23:26:34.167 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=started 2024-09-17T23:26:34.167 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_result 2024-09-17T23:26:34.167 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=stopped 2024-09-17T23:26:34.168 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_13 2024-09-17T23:26:34.168 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_instance=cluster1 2024-09-17T23:26:34.168 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:26:34.168 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ grep -i state 2024-09-17T23:26:34.168 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:26:34.168 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance=cluster1 2024-09-17T23:26:34.168 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local loop_instance 2024-09-17T23:26:34.168 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:26:34.168 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sleep 0 2024-09-17T23:26:34.168 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-17T23:26:34.170 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ seq 0 3 2024-09-17T23:26:34.171 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:26:34.171 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_13 2024-09-17T23:26:34.171 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:0 2024-09-17T23:26:34.171 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:26:34.171 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:26:34.171 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-17T23:26:34.171 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:0 2024-09-17T23:26:34.171 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:26:34.171 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:26:34.171 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:26:34.171 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=0 2024-09-17T23:26:34.171 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-17T23:26:34.171 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:26:34.172 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:26:34.172 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=0 2024-09-17T23:26:34.172 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:26:34.172 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-17T23:26:34.172 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:0 2024-09-17T23:26:34.172 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:26:34.172 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:26:34.172 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-17T23:26:34.172 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:0 2024-09-17T23:26:34.172 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:26:34.172 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:26:34.172 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:26:34.173 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:26:34.173 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-17T23:26:34.173 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:26:34.173 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:26:34.173 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=0 2024-09-17T23:26:34.173 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=0 2024-09-17T23:26:34.174 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-17T23:26:34.193 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:26:34.193 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:26:34.193 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:26:34.193 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_13 2024-09-17T23:26:34.318 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ return 0 2024-09-17T23:26:34.319 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ status_result=' "state": "Replaying"' 2024-09-17T23:26:34.319 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ echo ' "state": "Replaying"' 2024-09-17T23:26:34.319 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep -i Replaying 2024-09-17T23:26:34.320 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: "state": "Replaying" 2024-09-17T23:26:34.321 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_state=started 2024-09-17T23:26:34.321 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test started = started 2024-09-17T23:26:34.321 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:26:34.321 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_13 2024-09-17T23:26:34.321 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:26:34.321 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:26:34.321 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:26:34.322 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_13 2024-09-17T23:26:34.322 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = journal ']' 2024-09-17T23:26:34.322 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = snapshot ']' 2024-09-17T23:26:34.322 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_13 2024-09-17T23:26:34.322 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:26:34.322 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:26:34.322 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:26:34.322 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_13 2024-09-17T23:26:34.322 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster2-mirror-image_13.status 2024-09-17T23:26:34.322 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster2-mirror-image_13.status 2024-09-17T23:26:34.322 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:26:34.323 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_13.status 2024-09-17T23:26:34.323 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster1-mirror-image_13.status 2024-09-17T23:26:34.323 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster1-mirror-image_13.status 2024-09-17T23:26:34.323 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:26:34.324 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_status_log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_13.status 2024-09-17T23:26:34.324 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ mirror_image_snapshot cluster2 mirror image_13 2024-09-17T23:26:34.324 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:26:34.324 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:26:34.324 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_13 2024-09-17T23:26:34.324 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_13 2024-09-17T23:26:35.171 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Snapshot ID: 110 2024-09-17T23:26:35.184 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_13 /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_13.status 2024-09-17T23:26:35.185 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:26:35.185 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:26:35.185 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_13 2024-09-17T23:26:35.185 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_13.status 2024-09-17T23:26:35.185 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_13 --format xml 2024-09-17T23:26:35.185 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:26:35.233 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-17T23:26:35.235 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snapshot_id=110 2024-09-17T23:26:35.235 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:26:35.236 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:26:35.236 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.2 2024-09-17T23:26:35.437 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_13.status 2024-09-17T23:26:35.437 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:26:35.437 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:26:35.437 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_13 2024-09-17T23:26:35.438 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_13.status 2024-09-17T23:26:35.438 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2024-09-17T23:26:35.438 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:26:35.491 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:26:35.491 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:26:35.492 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:26:35.492 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:26:35.492 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:26:35.493 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:26:35.493 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 110 = '' 2024-09-17T23:26:35.493 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:26:35.493 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.4 2024-09-17T23:26:35.894 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_13.status 2024-09-17T23:26:35.895 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:26:35.895 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:26:35.895 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_13 2024-09-17T23:26:35.895 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_13.status 2024-09-17T23:26:35.895 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2024-09-17T23:26:35.895 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:26:35.947 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:26:35.947 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:26:35.949 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:26:35.949 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:26:35.949 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:26:35.950 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:26:35.950 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 110 = '' 2024-09-17T23:26:35.950 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:26:35.950 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.8 2024-09-17T23:26:36.241 INFO:tasks.ceph.cluster2.osd.2.smithi038.stderr:2024-09-17T23:26:36.234+0000 7f1c4b04a640 -1 reset not still connected to 0x561cc5dc8ea0 2024-09-17T23:26:36.241 INFO:tasks.ceph.cluster2.osd.2.smithi038.stderr:2024-09-17T23:26:36.234+0000 7f1c4b04a640 -1 reset not still connected to 0x561cc9894dd0 2024-09-17T23:26:36.241 INFO:tasks.ceph.cluster2.osd.2.smithi038.stderr:2024-09-17T23:26:36.234+0000 7f1c4b04a640 -1 reset not still connected to 0x561cce7af380 2024-09-17T23:26:36.241 INFO:tasks.ceph.cluster2.osd.2.smithi038.stderr:2024-09-17T23:26:36.234+0000 7f1c4b04a640 -1 reset not still connected to 0x561ccf7fe680 2024-09-17T23:26:36.245 INFO:tasks.ceph.cluster2.osd.2.smithi038.stderr:2024-09-17T23:26:36.243+0000 7f1c4b04a640 -1 reset not still connected to 0x561cd66725b0 2024-09-17T23:26:36.252 INFO:tasks.ceph.cluster2.osd.2.smithi038.stderr:2024-09-17T23:26:36.249+0000 7f1c4b04a640 -1 reset not still connected to 0x561cd80c6340 2024-09-17T23:26:36.252 INFO:tasks.ceph.cluster2.osd.2.smithi038.stderr:2024-09-17T23:26:36.249+0000 7f1c4b04a640 -1 reset not still connected to 0x561cd854ef70 2024-09-17T23:26:36.252 INFO:tasks.ceph.cluster2.osd.2.smithi038.stderr:2024-09-17T23:26:36.249+0000 7f1c4b04a640 -1 reset not still connected to 0x561cd89ba000 2024-09-17T23:26:36.252 INFO:tasks.ceph.cluster2.osd.2.smithi038.stderr:2024-09-17T23:26:36.249+0000 7f1c4b04a640 -1 reset not still connected to 0x561cd9975860 2024-09-17T23:26:36.252 INFO:tasks.ceph.cluster2.osd.2.smithi038.stderr:2024-09-17T23:26:36.249+0000 7f1c4b04a640 -1 reset not still connected to 0x561cda178340 2024-09-17T23:26:36.254 INFO:tasks.ceph.cluster2.osd.2.smithi038.stderr:2024-09-17T23:26:36.252+0000 7f1c4b04a640 -1 reset not still connected to 0x561ce5a8af70 2024-09-17T23:26:36.254 INFO:tasks.ceph.cluster2.osd.2.smithi038.stderr:2024-09-17T23:26:36.252+0000 7f1c4b04a640 -1 reset not still connected to 0x561ce9127a00 2024-09-17T23:26:36.254 INFO:tasks.ceph.cluster2.osd.2.smithi038.stderr:2024-09-17T23:26:36.252+0000 7f1c4b04a640 -1 reset not still connected to 0x561ce952b1e0 2024-09-17T23:26:36.254 INFO:tasks.ceph.cluster2.osd.2.smithi038.stderr:2024-09-17T23:26:36.252+0000 7f1c4b04a640 -1 reset not still connected to 0x561ce9ced110 2024-09-17T23:26:36.254 INFO:tasks.ceph.cluster2.osd.2.smithi038.stderr:2024-09-17T23:26:36.252+0000 7f1c4b04a640 -1 reset not still connected to 0x561cee725c70 2024-09-17T23:26:36.254 INFO:tasks.ceph.cluster2.osd.2.smithi038.stderr:2024-09-17T23:26:36.252+0000 7f1c4b04a640 -1 reset not still connected to 0x561cf092f860 2024-09-17T23:26:36.751 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_13.status 2024-09-17T23:26:36.751 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:26:36.751 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:26:36.751 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_13 2024-09-17T23:26:36.751 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_13.status 2024-09-17T23:26:36.752 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2024-09-17T23:26:36.752 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:26:36.800 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:26:36.802 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id=110 2024-09-17T23:26:36.802 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 110 = 110 2024-09-17T23:26:36.802 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:26:36.802 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snap_present cluster1 mirror image_13 snap 2024-09-17T23:26:36.802 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:26:36.802 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:26:36.802 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_13 2024-09-17T23:26:36.802 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:26:36.803 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:26:36.803 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-17T23:26:36.803 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:26:37.803 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 info mirror/image_13@snap 2024-09-17T23:26:37.852 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'image_13': 2024-09-17T23:26:37.853 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 128 MiB in 32 objects 2024-09-17T23:26:37.853 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:26:37.853 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 2 2024-09-17T23:26:37.853 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 10c6c9c1db6e 2024-09-17T23:26:37.853 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.10c6c9c1db6e 2024-09-17T23:26:37.853 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:26:37.853 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-17T23:26:37.853 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:26:37.853 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:26:37.853 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:18:19 2024 2024-09-17T23:26:37.853 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:18:19 2024 2024-09-17T23:26:37.853 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:18:19 2024 2024-09-17T23:26:37.854 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: protected: False 2024-09-17T23:26:37.854 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:26:37.854 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:26:37.854 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: 9ebe7c6f-3aa1-48f9-82b3-c20461625f84 2024-09-17T23:26:37.854 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: false 2024-09-17T23:26:37.856 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:26:37.856 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ compare_image_snaps mirror image_13 snap 2024-09-17T23:26:37.856 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:26:37.857 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_13 2024-09-17T23:26:37.857 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:26:37.857 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local ret=0 2024-09-17T23:26:37.857 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local rmt_export=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_13.export 2024-09-17T23:26:37.857 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local loc_export=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_13.export 2024-09-17T23:26:37.857 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_13.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_13.export 2024-09-17T23:26:37.857 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror export image_13@snap /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_13.export 2024-09-17T23:26:38.026 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-17T23:26:38.033 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 -p mirror export image_13@snap /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_13.export 2024-09-17T23:26:38.224 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-17T23:26:38.230 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ cmp /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_13.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_13.export 2024-09-17T23:26:38.293 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_13.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_13.export 2024-09-17T23:26:38.324 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:26:38.324 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:26:38.324 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_14 2024-09-17T23:26:38.325 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_snap cluster2 mirror image_14 snap 2024-09-17T23:26:38.325 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:26:38.325 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:26:38.325 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_14 2024-09-17T23:26:38.325 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:26:38.325 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror snap create image_14@snap --debug-rbd=20 --debug-journaler=20 2024-09-17T23:26:38.949 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2024-09-17T23:26:38.949 INFO:teuthology.orchestra.run:waiting for 600 2024-09-17T23:26:38.949 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2024-09-17T23:26:38.949 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2024-09-17T23:26:38.950 DEBUG:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2024-09-17T23:26:38.953 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2024-09-17T23:26:38.953 INFO:tasks.rbd_mirror_thrash:waiting for 79 secs before thrashing 2024-09-17T23:26:40.196 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_started cluster1 mirror image_14 2024-09-17T23:26:40.225 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:26:40.225 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:26:40.225 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_14 2024-09-17T23:26:40.225 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_state cluster1 mirror image_14 started 2024-09-17T23:26:40.226 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:26:40.226 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:26:40.226 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_14 2024-09-17T23:26:40.226 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=started 2024-09-17T23:26:40.226 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:26:40.226 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:26:40.226 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:26:41.197 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_replay_state cluster1 mirror image_14 started 2024-09-17T23:26:41.197 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:26:41.198 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:26:41.198 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_14 2024-09-17T23:26:41.198 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=started 2024-09-17T23:26:41.198 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_result 2024-09-17T23:26:41.198 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=stopped 2024-09-17T23:26:41.199 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_14 2024-09-17T23:26:41.199 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_instance=cluster1 2024-09-17T23:26:41.199 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ grep -i state 2024-09-17T23:26:41.199 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:26:41.199 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:26:41.199 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance=cluster1 2024-09-17T23:26:41.200 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local loop_instance 2024-09-17T23:26:41.200 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:26:41.200 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sleep 0 2024-09-17T23:26:41.200 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-17T23:26:41.200 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ seq 0 3 2024-09-17T23:26:41.202 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:26:41.202 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_14 2024-09-17T23:26:41.202 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:0 2024-09-17T23:26:41.202 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:26:41.202 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:26:41.202 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-17T23:26:41.202 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:0 2024-09-17T23:26:41.202 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:26:41.203 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:26:41.203 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:26:41.203 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=0 2024-09-17T23:26:41.203 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-17T23:26:41.203 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:26:41.203 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:26:41.203 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=0 2024-09-17T23:26:41.203 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:26:41.203 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-17T23:26:41.203 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:0 2024-09-17T23:26:41.203 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:26:41.203 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:26:41.203 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-17T23:26:41.204 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:0 2024-09-17T23:26:41.205 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:26:41.205 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:26:41.205 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:26:41.205 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:26:41.205 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-17T23:26:41.205 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:26:41.205 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:26:41.205 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=0 2024-09-17T23:26:41.205 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=0 2024-09-17T23:26:41.205 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-17T23:26:41.224 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:26:41.224 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:26:41.224 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:26:41.224 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_14 2024-09-17T23:26:41.348 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ return 0 2024-09-17T23:26:41.349 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ status_result=' "state": "Replaying"' 2024-09-17T23:26:41.349 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ echo ' "state": "Replaying"' 2024-09-17T23:26:41.349 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep -i Replaying 2024-09-17T23:26:41.350 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: "state": "Replaying" 2024-09-17T23:26:41.351 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_state=started 2024-09-17T23:26:41.351 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test started = started 2024-09-17T23:26:41.351 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:26:41.351 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_14 2024-09-17T23:26:41.352 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:26:41.352 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:26:41.352 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:26:41.352 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_14 2024-09-17T23:26:41.352 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = journal ']' 2024-09-17T23:26:41.352 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = snapshot ']' 2024-09-17T23:26:41.352 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_14 2024-09-17T23:26:41.352 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:26:41.352 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:26:41.352 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:26:41.352 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_14 2024-09-17T23:26:41.352 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster2-mirror-image_14.status 2024-09-17T23:26:41.352 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster2-mirror-image_14.status 2024-09-17T23:26:41.353 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:26:41.353 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_14.status 2024-09-17T23:26:41.353 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster1-mirror-image_14.status 2024-09-17T23:26:41.353 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster1-mirror-image_14.status 2024-09-17T23:26:41.353 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:26:41.354 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_status_log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_14.status 2024-09-17T23:26:41.354 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ mirror_image_snapshot cluster2 mirror image_14 2024-09-17T23:26:41.354 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:26:41.354 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:26:41.355 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_14 2024-09-17T23:26:41.355 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_14 2024-09-17T23:26:42.206 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Snapshot ID: 112 2024-09-17T23:26:42.219 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_14 /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_14.status 2024-09-17T23:26:42.219 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:26:42.219 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:26:42.219 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_14 2024-09-17T23:26:42.219 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_14.status 2024-09-17T23:26:42.219 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_14 --format xml 2024-09-17T23:26:42.219 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:26:42.267 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-17T23:26:42.269 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snapshot_id=112 2024-09-17T23:26:42.269 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:26:42.269 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:26:42.269 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.2 2024-09-17T23:26:42.470 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_14.status 2024-09-17T23:26:42.470 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:26:42.470 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:26:42.470 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_14 2024-09-17T23:26:42.471 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_14.status 2024-09-17T23:26:42.471 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2024-09-17T23:26:42.471 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:26:42.522 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:26:42.522 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:26:42.524 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:26:42.524 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:26:42.524 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:26:42.524 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:26:42.524 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 112 = '' 2024-09-17T23:26:42.525 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:26:42.525 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.4 2024-09-17T23:26:42.926 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_14.status 2024-09-17T23:26:42.926 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:26:42.926 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:26:42.926 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_14 2024-09-17T23:26:42.926 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_14.status 2024-09-17T23:26:42.926 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2024-09-17T23:26:42.926 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:26:42.980 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:26:42.980 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:26:42.981 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:26:42.981 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:26:42.981 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:26:42.982 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:26:42.982 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 112 = '' 2024-09-17T23:26:42.982 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:26:42.982 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.8 2024-09-17T23:26:43.783 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_14.status 2024-09-17T23:26:43.783 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:26:43.784 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:26:43.784 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_14 2024-09-17T23:26:43.784 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_14.status 2024-09-17T23:26:43.784 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2024-09-17T23:26:43.784 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:26:43.832 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:26:43.834 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id=112 2024-09-17T23:26:43.834 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 112 = 112 2024-09-17T23:26:43.834 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:26:43.834 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snap_present cluster1 mirror image_14 snap 2024-09-17T23:26:43.834 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:26:43.834 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:26:43.834 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_14 2024-09-17T23:26:43.835 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:26:43.835 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:26:43.835 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-17T23:26:43.835 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:26:44.835 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 info mirror/image_14@snap 2024-09-17T23:26:44.883 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'image_14': 2024-09-17T23:26:44.883 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 128 MiB in 32 objects 2024-09-17T23:26:44.883 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:26:44.883 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 2 2024-09-17T23:26:44.884 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 10c6ba4a7425 2024-09-17T23:26:44.884 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.10c6ba4a7425 2024-09-17T23:26:44.884 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:26:44.884 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-17T23:26:44.884 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:26:44.884 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:26:44.884 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:18:31 2024 2024-09-17T23:26:44.884 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:18:31 2024 2024-09-17T23:26:44.884 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:18:31 2024 2024-09-17T23:26:44.884 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: protected: False 2024-09-17T23:26:44.884 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:26:44.884 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:26:44.884 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: 115aba0c-a419-49de-b089-0afc43656b9c 2024-09-17T23:26:44.885 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: false 2024-09-17T23:26:44.887 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:26:44.887 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ compare_image_snaps mirror image_14 snap 2024-09-17T23:26:44.887 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:26:44.887 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_14 2024-09-17T23:26:44.888 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:26:44.888 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local ret=0 2024-09-17T23:26:44.888 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local rmt_export=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_14.export 2024-09-17T23:26:44.888 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local loc_export=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_14.export 2024-09-17T23:26:44.888 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_14.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_14.export 2024-09-17T23:26:44.888 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror export image_14@snap /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_14.export 2024-09-17T23:26:45.055 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-17T23:26:45.063 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 -p mirror export image_14@snap /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_14.export 2024-09-17T23:26:45.273 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-17T23:26:45.280 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ cmp /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_14.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_14.export 2024-09-17T23:26:45.342 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_14.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_14.export 2024-09-17T23:26:45.373 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:26:45.373 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:26:45.373 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_15 2024-09-17T23:26:45.373 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_snap cluster2 mirror image_15 snap 2024-09-17T23:26:45.373 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:26:45.374 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:26:45.374 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_15 2024-09-17T23:26:45.374 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:26:45.374 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror snap create image_15@snap --debug-rbd=20 --debug-journaler=20 2024-09-17T23:26:47.234 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_started cluster1 mirror image_15 2024-09-17T23:26:47.235 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:26:47.235 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:26:47.235 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_15 2024-09-17T23:26:47.235 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_state cluster1 mirror image_15 started 2024-09-17T23:26:47.235 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:26:47.235 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:26:47.235 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_15 2024-09-17T23:26:47.235 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=started 2024-09-17T23:26:47.235 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:26:47.235 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:26:47.235 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:26:48.236 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_replay_state cluster1 mirror image_15 started 2024-09-17T23:26:48.236 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:26:48.236 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:26:48.236 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_15 2024-09-17T23:26:48.236 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=started 2024-09-17T23:26:48.236 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_result 2024-09-17T23:26:48.236 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=stopped 2024-09-17T23:26:48.237 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_15 2024-09-17T23:26:48.237 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_instance=cluster1 2024-09-17T23:26:48.237 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:26:48.237 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ grep -i state 2024-09-17T23:26:48.237 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:26:48.238 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance=cluster1 2024-09-17T23:26:48.238 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local loop_instance 2024-09-17T23:26:48.238 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:26:48.238 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sleep 0 2024-09-17T23:26:48.238 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-17T23:26:48.238 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ seq 0 3 2024-09-17T23:26:48.240 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:26:48.240 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_15 2024-09-17T23:26:48.240 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:0 2024-09-17T23:26:48.240 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:26:48.240 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:26:48.241 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-17T23:26:48.241 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:0 2024-09-17T23:26:48.241 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:26:48.241 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:26:48.241 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:26:48.241 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=0 2024-09-17T23:26:48.241 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-17T23:26:48.241 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:26:48.241 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:26:48.241 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=0 2024-09-17T23:26:48.241 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:26:48.241 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-17T23:26:48.241 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:0 2024-09-17T23:26:48.242 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:26:48.242 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:26:48.242 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-17T23:26:48.242 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:0 2024-09-17T23:26:48.243 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:26:48.243 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:26:48.243 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:26:48.243 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:26:48.243 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-17T23:26:48.243 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:26:48.243 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:26:48.243 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=0 2024-09-17T23:26:48.243 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=0 2024-09-17T23:26:48.243 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-17T23:26:48.264 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:26:48.264 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:26:48.264 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:26:48.264 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_15 2024-09-17T23:26:48.390 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ return 0 2024-09-17T23:26:48.391 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ status_result=' "state": "Replaying"' 2024-09-17T23:26:48.391 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ echo ' "state": "Replaying"' 2024-09-17T23:26:48.391 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep -i Replaying 2024-09-17T23:26:48.392 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: "state": "Replaying" 2024-09-17T23:26:48.393 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_state=started 2024-09-17T23:26:48.393 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test started = started 2024-09-17T23:26:48.394 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:26:48.394 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_15 2024-09-17T23:26:48.394 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:26:48.394 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:26:48.394 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:26:48.394 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_15 2024-09-17T23:26:48.394 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = journal ']' 2024-09-17T23:26:48.394 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = snapshot ']' 2024-09-17T23:26:48.394 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_15 2024-09-17T23:26:48.394 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:26:48.394 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:26:48.394 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:26:48.394 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_15 2024-09-17T23:26:48.394 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster2-mirror-image_15.status 2024-09-17T23:26:48.395 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster2-mirror-image_15.status 2024-09-17T23:26:48.395 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:26:48.395 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_15.status 2024-09-17T23:26:48.395 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster1-mirror-image_15.status 2024-09-17T23:26:48.395 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster1-mirror-image_15.status 2024-09-17T23:26:48.395 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:26:48.396 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_status_log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_15.status 2024-09-17T23:26:48.396 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ mirror_image_snapshot cluster2 mirror image_15 2024-09-17T23:26:48.396 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:26:48.396 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:26:48.396 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_15 2024-09-17T23:26:48.396 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_15 2024-09-17T23:26:49.233 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Snapshot ID: 114 2024-09-17T23:26:49.244 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_15 /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_15.status 2024-09-17T23:26:49.244 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:26:49.245 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:26:49.245 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_15 2024-09-17T23:26:49.245 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_15.status 2024-09-17T23:26:49.245 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_15 --format xml 2024-09-17T23:26:49.245 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:26:49.291 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-17T23:26:49.293 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snapshot_id=114 2024-09-17T23:26:49.293 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:26:49.293 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:26:49.293 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.2 2024-09-17T23:26:49.495 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_15.status 2024-09-17T23:26:49.495 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:26:49.495 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:26:49.495 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_15 2024-09-17T23:26:49.495 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_15.status 2024-09-17T23:26:49.495 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2024-09-17T23:26:49.495 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:26:49.551 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:26:49.551 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:26:49.553 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:26:49.553 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:26:49.553 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:26:49.553 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:26:49.554 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 114 = '' 2024-09-17T23:26:49.554 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:26:49.554 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.4 2024-09-17T23:26:49.955 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_15.status 2024-09-17T23:26:49.955 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:26:49.955 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:26:49.955 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_15 2024-09-17T23:26:49.955 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_15.status 2024-09-17T23:26:49.955 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2024-09-17T23:26:49.956 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:26:50.007 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:26:50.008 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:26:50.009 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:26:50.009 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:26:50.009 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:26:50.010 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:26:50.010 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 114 = '' 2024-09-17T23:26:50.010 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:26:50.010 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.8 2024-09-17T23:26:50.811 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_15.status 2024-09-17T23:26:50.812 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:26:50.812 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:26:50.812 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_15 2024-09-17T23:26:50.812 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_15.status 2024-09-17T23:26:50.812 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2024-09-17T23:26:50.812 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:26:50.862 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:26:50.864 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id=114 2024-09-17T23:26:50.864 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 114 = 114 2024-09-17T23:26:50.864 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:26:50.864 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snap_present cluster1 mirror image_15 snap 2024-09-17T23:26:50.864 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:26:50.864 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:26:50.864 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_15 2024-09-17T23:26:50.864 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:26:50.865 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:26:50.865 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-17T23:26:50.865 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:26:51.326 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-17T23:26:51.323+0000 7f0394bcc640 -1 reset not still connected to 0x56053cdfef70 2024-09-17T23:26:51.326 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-17T23:26:51.324+0000 7f0394bcc640 -1 reset not still connected to 0x56053d24add0 2024-09-17T23:26:51.326 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-17T23:26:51.324+0000 7f0394bcc640 -1 reset not still connected to 0x56053d51edd0 2024-09-17T23:26:51.326 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-17T23:26:51.324+0000 7f0394bcc640 -1 reset not still connected to 0x56053d6a68f0 2024-09-17T23:26:51.327 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-17T23:26:51.324+0000 7f0394bcc640 -1 reset not still connected to 0x56053e4ab6c0 2024-09-17T23:26:51.333 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-17T23:26:51.331+0000 7f0394bcc640 -1 reset not still connected to 0x56053f189c70 2024-09-17T23:26:51.333 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-17T23:26:51.331+0000 7f0394bcc640 -1 reset not still connected to 0x56053f2fe680 2024-09-17T23:26:51.333 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-17T23:26:51.331+0000 7f0394bcc640 -1 reset not still connected to 0x56053fd1e410 2024-09-17T23:26:51.333 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-17T23:26:51.331+0000 7f0394bcc640 -1 reset not still connected to 0x56053fd70750 2024-09-17T23:26:51.865 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 info mirror/image_15@snap 2024-09-17T23:26:51.915 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'image_15': 2024-09-17T23:26:51.915 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 128 MiB in 32 objects 2024-09-17T23:26:51.915 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:26:51.915 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 2 2024-09-17T23:26:51.915 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 10c629418b51 2024-09-17T23:26:51.915 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.10c629418b51 2024-09-17T23:26:51.916 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:26:51.916 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-17T23:26:51.916 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:26:51.916 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:26:51.916 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:18:42 2024 2024-09-17T23:26:51.916 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:18:42 2024 2024-09-17T23:26:51.916 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:18:42 2024 2024-09-17T23:26:51.916 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: protected: False 2024-09-17T23:26:51.916 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:26:51.916 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:26:51.917 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: c5adf338-7077-4024-ab02-6a785842239c 2024-09-17T23:26:51.917 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: false 2024-09-17T23:26:51.919 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:26:51.919 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ compare_image_snaps mirror image_15 snap 2024-09-17T23:26:51.919 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:26:51.919 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_15 2024-09-17T23:26:51.919 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:26:51.920 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local ret=0 2024-09-17T23:26:51.920 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local rmt_export=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_15.export 2024-09-17T23:26:51.920 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local loc_export=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_15.export 2024-09-17T23:26:51.920 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_15.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_15.export 2024-09-17T23:26:51.920 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror export image_15@snap /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_15.export 2024-09-17T23:26:52.096 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-17T23:26:52.104 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 -p mirror export image_15@snap /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_15.export 2024-09-17T23:26:52.341 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-17T23:26:52.348 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ cmp /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_15.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_15.export 2024-09-17T23:26:52.410 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_15.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_15.export 2024-09-17T23:26:52.442 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:26:52.442 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:26:52.442 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_16 2024-09-17T23:26:52.442 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_snap cluster2 mirror image_16 snap 2024-09-17T23:26:52.442 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:26:52.442 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:26:52.442 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_16 2024-09-17T23:26:52.442 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:26:52.442 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror snap create image_16@snap --debug-rbd=20 --debug-journaler=20 2024-09-17T23:26:54.257 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_started cluster1 mirror image_16 2024-09-17T23:26:54.258 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:26:54.258 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:26:54.258 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_16 2024-09-17T23:26:54.258 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_state cluster1 mirror image_16 started 2024-09-17T23:26:54.258 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:26:54.258 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:26:54.258 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_16 2024-09-17T23:26:54.258 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=started 2024-09-17T23:26:54.258 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:26:54.258 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:26:54.258 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:26:55.259 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_replay_state cluster1 mirror image_16 started 2024-09-17T23:26:55.259 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:26:55.259 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:26:55.259 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_16 2024-09-17T23:26:55.259 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=started 2024-09-17T23:26:55.259 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_result 2024-09-17T23:26:55.259 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=stopped 2024-09-17T23:26:55.260 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_16 2024-09-17T23:26:55.260 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ grep -i state 2024-09-17T23:26:55.260 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_instance=cluster1 2024-09-17T23:26:55.260 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:26:55.260 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:26:55.261 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance=cluster1 2024-09-17T23:26:55.261 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local loop_instance 2024-09-17T23:26:55.261 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:26:55.261 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sleep 0 2024-09-17T23:26:55.261 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-17T23:26:55.261 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ seq 0 3 2024-09-17T23:26:55.263 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:26:55.263 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_16 2024-09-17T23:26:55.263 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:0 2024-09-17T23:26:55.263 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:26:55.263 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:26:55.263 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-17T23:26:55.263 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:0 2024-09-17T23:26:55.263 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:26:55.263 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:26:55.264 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:26:55.264 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=0 2024-09-17T23:26:55.264 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-17T23:26:55.264 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:26:55.264 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:26:55.264 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=0 2024-09-17T23:26:55.264 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:26:55.264 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-17T23:26:55.264 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:0 2024-09-17T23:26:55.265 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:26:55.265 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:26:55.265 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-17T23:26:55.265 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:0 2024-09-17T23:26:55.265 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:26:55.265 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:26:55.266 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:26:55.266 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:26:55.266 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-17T23:26:55.266 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:26:55.266 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:26:55.266 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=0 2024-09-17T23:26:55.266 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=0 2024-09-17T23:26:55.266 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-17T23:26:55.287 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:26:55.287 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:26:55.287 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:26:55.287 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_16 2024-09-17T23:26:55.412 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ return 0 2024-09-17T23:26:55.412 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ status_result=' "state": "Replaying"' 2024-09-17T23:26:55.413 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ echo ' "state": "Replaying"' 2024-09-17T23:26:55.413 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep -i Replaying 2024-09-17T23:26:55.413 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: "state": "Replaying" 2024-09-17T23:26:55.415 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_state=started 2024-09-17T23:26:55.415 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test started = started 2024-09-17T23:26:55.415 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:26:55.415 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_16 2024-09-17T23:26:55.415 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:26:55.415 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:26:55.415 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:26:55.415 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_16 2024-09-17T23:26:55.415 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = journal ']' 2024-09-17T23:26:55.415 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = snapshot ']' 2024-09-17T23:26:55.415 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_16 2024-09-17T23:26:55.415 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:26:55.415 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:26:55.415 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:26:55.416 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_16 2024-09-17T23:26:55.416 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster2-mirror-image_16.status 2024-09-17T23:26:55.416 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster2-mirror-image_16.status 2024-09-17T23:26:55.416 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:26:55.416 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_16.status 2024-09-17T23:26:55.417 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster1-mirror-image_16.status 2024-09-17T23:26:55.417 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster1-mirror-image_16.status 2024-09-17T23:26:55.417 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:26:55.418 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_status_log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_16.status 2024-09-17T23:26:55.418 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ mirror_image_snapshot cluster2 mirror image_16 2024-09-17T23:26:55.418 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:26:55.418 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:26:55.418 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_16 2024-09-17T23:26:55.418 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_16 2024-09-17T23:26:56.266 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Snapshot ID: 116 2024-09-17T23:26:56.278 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_16 /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_16.status 2024-09-17T23:26:56.278 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:26:56.278 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:26:56.278 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_16 2024-09-17T23:26:56.279 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_16.status 2024-09-17T23:26:56.279 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_16 --format xml 2024-09-17T23:26:56.279 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:26:56.328 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-17T23:26:56.330 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snapshot_id=116 2024-09-17T23:26:56.330 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:26:56.330 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:26:56.330 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.2 2024-09-17T23:26:56.532 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_16.status 2024-09-17T23:26:56.532 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:26:56.532 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:26:56.532 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_16 2024-09-17T23:26:56.532 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_16.status 2024-09-17T23:26:56.532 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2024-09-17T23:26:56.532 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:26:56.588 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:26:56.588 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:26:56.590 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:26:56.590 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:26:56.590 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:26:56.590 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:26:56.590 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 116 = '' 2024-09-17T23:26:56.590 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:26:56.591 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.4 2024-09-17T23:26:56.992 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_16.status 2024-09-17T23:26:56.992 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:26:56.992 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:26:56.992 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_16 2024-09-17T23:26:56.992 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_16.status 2024-09-17T23:26:56.992 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2024-09-17T23:26:56.992 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:26:57.041 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:26:57.041 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:26:57.042 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:26:57.042 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:26:57.043 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:26:57.043 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:26:57.043 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 116 = '' 2024-09-17T23:26:57.043 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:26:57.043 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.8 2024-09-17T23:26:57.844 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_16.status 2024-09-17T23:26:57.845 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:26:57.845 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:26:57.845 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_16 2024-09-17T23:26:57.845 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_16.status 2024-09-17T23:26:57.845 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2024-09-17T23:26:57.845 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:26:57.892 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:26:57.895 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id=116 2024-09-17T23:26:57.895 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 116 = 116 2024-09-17T23:26:57.895 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:26:57.895 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snap_present cluster1 mirror image_16 snap 2024-09-17T23:26:57.895 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:26:57.895 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:26:57.895 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_16 2024-09-17T23:26:57.895 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:26:57.895 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:26:57.895 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-17T23:26:57.895 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:26:58.896 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 info mirror/image_16@snap 2024-09-17T23:26:58.943 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'image_16': 2024-09-17T23:26:58.943 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 128 MiB in 32 objects 2024-09-17T23:26:58.943 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:26:58.943 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 2 2024-09-17T23:26:58.943 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 10c68b4b411d 2024-09-17T23:26:58.943 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.10c68b4b411d 2024-09-17T23:26:58.944 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:26:58.944 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-17T23:26:58.944 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:26:58.944 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:26:58.944 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:18:55 2024 2024-09-17T23:26:58.944 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:18:55 2024 2024-09-17T23:26:58.944 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:18:55 2024 2024-09-17T23:26:58.944 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: protected: False 2024-09-17T23:26:58.944 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:26:58.944 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:26:58.944 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: b0d8f9e8-4e5b-45cd-b488-325c8908f543 2024-09-17T23:26:58.944 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: false 2024-09-17T23:26:58.947 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:26:58.947 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ compare_image_snaps mirror image_16 snap 2024-09-17T23:26:58.947 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:26:58.947 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_16 2024-09-17T23:26:58.947 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:26:58.947 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local ret=0 2024-09-17T23:26:58.947 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local rmt_export=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_16.export 2024-09-17T23:26:58.948 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local loc_export=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_16.export 2024-09-17T23:26:58.948 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_16.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_16.export 2024-09-17T23:26:58.948 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror export image_16@snap /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_16.export 2024-09-17T23:26:59.115 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-17T23:26:59.121 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 -p mirror export image_16@snap /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_16.export 2024-09-17T23:26:59.324 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-17T23:26:59.331 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ cmp /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_16.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_16.export 2024-09-17T23:26:59.393 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_16.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_16.export 2024-09-17T23:26:59.424 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:26:59.424 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:26:59.424 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_17 2024-09-17T23:26:59.424 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_snap cluster2 mirror image_17 snap 2024-09-17T23:26:59.424 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:26:59.424 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:26:59.424 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_17 2024-09-17T23:26:59.424 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:26:59.424 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror snap create image_17@snap --debug-rbd=20 --debug-journaler=20 2024-09-17T23:27:01.304 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_started cluster1 mirror image_17 2024-09-17T23:27:01.304 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:27:01.304 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:27:01.304 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_17 2024-09-17T23:27:01.304 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_state cluster1 mirror image_17 started 2024-09-17T23:27:01.305 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:27:01.305 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:27:01.305 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_17 2024-09-17T23:27:01.305 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=started 2024-09-17T23:27:01.305 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:27:01.305 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:27:01.305 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:27:02.305 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_replay_state cluster1 mirror image_17 started 2024-09-17T23:27:02.305 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:27:02.306 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:27:02.306 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_17 2024-09-17T23:27:02.306 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=started 2024-09-17T23:27:02.306 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_result 2024-09-17T23:27:02.306 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=stopped 2024-09-17T23:27:02.307 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_17 2024-09-17T23:27:02.307 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ grep -i state 2024-09-17T23:27:02.307 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_instance=cluster1 2024-09-17T23:27:02.307 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:27:02.307 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:27:02.307 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance=cluster1 2024-09-17T23:27:02.308 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local loop_instance 2024-09-17T23:27:02.308 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:27:02.308 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sleep 0 2024-09-17T23:27:02.308 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-17T23:27:02.309 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ seq 0 3 2024-09-17T23:27:02.310 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:27:02.310 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_17 2024-09-17T23:27:02.310 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:0 2024-09-17T23:27:02.310 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:27:02.310 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:27:02.310 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-17T23:27:02.310 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:0 2024-09-17T23:27:02.310 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:27:02.310 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:27:02.310 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:27:02.310 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=0 2024-09-17T23:27:02.310 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-17T23:27:02.310 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:27:02.311 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:27:02.311 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=0 2024-09-17T23:27:02.311 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:27:02.311 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-17T23:27:02.312 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:0 2024-09-17T23:27:02.312 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:27:02.313 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:27:02.313 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-17T23:27:02.313 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:0 2024-09-17T23:27:02.313 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:27:02.313 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:27:02.313 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:27:02.313 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:27:02.313 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-17T23:27:02.313 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:27:02.313 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:27:02.313 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=0 2024-09-17T23:27:02.314 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=0 2024-09-17T23:27:02.314 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-17T23:27:02.332 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:27:02.333 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:27:02.333 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:27:02.333 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_17 2024-09-17T23:27:02.457 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ return 0 2024-09-17T23:27:02.458 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ status_result=' "state": "Replaying"' 2024-09-17T23:27:02.458 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ echo ' "state": "Replaying"' 2024-09-17T23:27:02.458 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep -i Replaying 2024-09-17T23:27:02.459 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: "state": "Replaying" 2024-09-17T23:27:02.461 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_state=started 2024-09-17T23:27:02.461 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test started = started 2024-09-17T23:27:02.461 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:27:02.461 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_17 2024-09-17T23:27:02.461 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:27:02.461 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:27:02.461 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:27:02.461 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_17 2024-09-17T23:27:02.461 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = journal ']' 2024-09-17T23:27:02.462 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = snapshot ']' 2024-09-17T23:27:02.462 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_17 2024-09-17T23:27:02.462 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:27:02.462 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:27:02.462 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:27:02.462 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_17 2024-09-17T23:27:02.462 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster2-mirror-image_17.status 2024-09-17T23:27:02.462 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster2-mirror-image_17.status 2024-09-17T23:27:02.462 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:27:02.464 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_17.status 2024-09-17T23:27:02.464 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster1-mirror-image_17.status 2024-09-17T23:27:02.464 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster1-mirror-image_17.status 2024-09-17T23:27:02.464 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:27:02.464 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_status_log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_17.status 2024-09-17T23:27:02.464 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ mirror_image_snapshot cluster2 mirror image_17 2024-09-17T23:27:02.464 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:27:02.464 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:27:02.464 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_17 2024-09-17T23:27:02.464 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_17 2024-09-17T23:27:03.310 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Snapshot ID: 118 2024-09-17T23:27:03.322 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_17 /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_17.status 2024-09-17T23:27:03.322 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:27:03.322 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:27:03.322 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_17 2024-09-17T23:27:03.322 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_17.status 2024-09-17T23:27:03.322 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_17 --format xml 2024-09-17T23:27:03.322 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:27:03.370 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-17T23:27:03.372 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snapshot_id=118 2024-09-17T23:27:03.372 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:27:03.372 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:27:03.372 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.2 2024-09-17T23:27:03.574 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_17.status 2024-09-17T23:27:03.574 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:27:03.574 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:27:03.574 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_17 2024-09-17T23:27:03.574 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_17.status 2024-09-17T23:27:03.574 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2024-09-17T23:27:03.574 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:27:03.626 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:27:03.628 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id=51 2024-09-17T23:27:03.628 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 118 = 51 2024-09-17T23:27:03.628 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:27:03.628 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.4 2024-09-17T23:27:04.030 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_17.status 2024-09-17T23:27:04.030 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:27:04.030 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:27:04.030 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_17 2024-09-17T23:27:04.030 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_17.status 2024-09-17T23:27:04.030 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2024-09-17T23:27:04.030 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:27:04.084 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:27:04.086 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id=51 2024-09-17T23:27:04.086 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 118 = 51 2024-09-17T23:27:04.086 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:27:04.086 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.8 2024-09-17T23:27:04.888 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_17.status 2024-09-17T23:27:04.888 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:27:04.888 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:27:04.888 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_17 2024-09-17T23:27:04.888 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_17.status 2024-09-17T23:27:04.888 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2024-09-17T23:27:04.888 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:27:04.938 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:27:04.939 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:27:04.940 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:27:04.940 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:27:04.940 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:27:04.941 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:27:04.941 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 118 = '' 2024-09-17T23:27:04.941 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:27:04.941 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1.6 2024-09-17T23:27:06.542 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_17.status 2024-09-17T23:27:06.542 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:27:06.543 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:27:06.543 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_17 2024-09-17T23:27:06.543 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_17.status 2024-09-17T23:27:06.543 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2024-09-17T23:27:06.543 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:27:06.598 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:27:06.601 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id=118 2024-09-17T23:27:06.601 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 118 = 118 2024-09-17T23:27:06.601 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:27:06.601 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snap_present cluster1 mirror image_17 snap 2024-09-17T23:27:06.601 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:27:06.601 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:27:06.601 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_17 2024-09-17T23:27:06.601 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:27:06.601 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:27:06.601 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-17T23:27:06.601 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:27:07.602 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 info mirror/image_17@snap 2024-09-17T23:27:07.651 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'image_17': 2024-09-17T23:27:07.651 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 128 MiB in 32 objects 2024-09-17T23:27:07.651 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:27:07.651 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 2 2024-09-17T23:27:07.651 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 10c6acb536b0 2024-09-17T23:27:07.651 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.10c6acb536b0 2024-09-17T23:27:07.651 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:27:07.652 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-17T23:27:07.652 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:27:07.652 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:27:07.652 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:19:10 2024 2024-09-17T23:27:07.652 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:19:10 2024 2024-09-17T23:27:07.652 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:19:10 2024 2024-09-17T23:27:07.652 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: protected: False 2024-09-17T23:27:07.652 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:27:07.652 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:27:07.652 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: 1c42b8ab-b2d2-4388-b7b5-b3e796c582f2 2024-09-17T23:27:07.652 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: false 2024-09-17T23:27:07.655 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:27:07.655 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ compare_image_snaps mirror image_17 snap 2024-09-17T23:27:07.655 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:27:07.656 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_17 2024-09-17T23:27:07.656 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:27:07.656 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local ret=0 2024-09-17T23:27:07.656 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local rmt_export=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_17.export 2024-09-17T23:27:07.656 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local loc_export=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_17.export 2024-09-17T23:27:07.656 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_17.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_17.export 2024-09-17T23:27:07.656 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror export image_17@snap /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_17.export 2024-09-17T23:27:07.827 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-17T23:27:07.834 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 -p mirror export image_17@snap /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_17.export 2024-09-17T23:27:08.074 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-17T23:27:08.080 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ cmp /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_17.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_17.export 2024-09-17T23:27:08.143 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_17.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_17.export 2024-09-17T23:27:08.174 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:27:08.174 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:27:08.174 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_18 2024-09-17T23:27:08.174 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_snap cluster2 mirror image_18 snap 2024-09-17T23:27:08.174 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:27:08.174 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:27:08.174 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_18 2024-09-17T23:27:08.174 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:27:08.174 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror snap create image_18@snap --debug-rbd=20 --debug-journaler=20 2024-09-17T23:27:09.957 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_started cluster1 mirror image_18 2024-09-17T23:27:09.957 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:27:09.957 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:27:09.957 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_18 2024-09-17T23:27:09.957 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_state cluster1 mirror image_18 started 2024-09-17T23:27:09.957 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:27:09.957 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:27:09.957 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_18 2024-09-17T23:27:09.957 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=started 2024-09-17T23:27:09.958 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:27:09.958 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:27:09.958 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:27:10.958 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_replay_state cluster1 mirror image_18 started 2024-09-17T23:27:10.958 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:27:10.958 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:27:10.958 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_18 2024-09-17T23:27:10.958 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=started 2024-09-17T23:27:10.958 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_result 2024-09-17T23:27:10.958 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=stopped 2024-09-17T23:27:10.959 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_18 2024-09-17T23:27:10.959 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_instance=cluster1 2024-09-17T23:27:10.959 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:27:10.959 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ grep -i state 2024-09-17T23:27:10.960 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:27:10.960 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance=cluster1 2024-09-17T23:27:10.960 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local loop_instance 2024-09-17T23:27:10.960 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:27:10.960 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sleep 0 2024-09-17T23:27:10.960 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-17T23:27:10.960 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ seq 0 3 2024-09-17T23:27:10.963 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:27:10.963 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_18 2024-09-17T23:27:10.963 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:0 2024-09-17T23:27:10.963 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:27:10.963 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:27:10.963 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-17T23:27:10.963 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:0 2024-09-17T23:27:10.963 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:27:10.963 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:27:10.963 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:27:10.964 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=0 2024-09-17T23:27:10.964 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-17T23:27:10.964 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:27:10.964 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:27:10.964 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=0 2024-09-17T23:27:10.964 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:27:10.964 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-17T23:27:10.964 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:0 2024-09-17T23:27:10.964 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:27:10.964 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:27:10.964 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-17T23:27:10.964 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:0 2024-09-17T23:27:10.964 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:27:10.964 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:27:10.965 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:27:10.965 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:27:10.965 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-17T23:27:10.965 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:27:10.965 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:27:10.965 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=0 2024-09-17T23:27:10.965 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=0 2024-09-17T23:27:10.965 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-17T23:27:10.985 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:27:10.985 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:27:10.985 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:27:10.985 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_18 2024-09-17T23:27:11.113 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ return 0 2024-09-17T23:27:11.114 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ status_result=' "state": "Replaying"' 2024-09-17T23:27:11.114 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ echo ' "state": "Replaying"' 2024-09-17T23:27:11.114 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep -i Replaying 2024-09-17T23:27:11.115 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: "state": "Replaying" 2024-09-17T23:27:11.116 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_state=started 2024-09-17T23:27:11.117 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test started = started 2024-09-17T23:27:11.117 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:27:11.117 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_18 2024-09-17T23:27:11.117 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:27:11.117 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:27:11.117 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:27:11.117 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_18 2024-09-17T23:27:11.117 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = journal ']' 2024-09-17T23:27:11.117 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = snapshot ']' 2024-09-17T23:27:11.117 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_18 2024-09-17T23:27:11.117 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:27:11.117 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:27:11.117 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:27:11.117 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_18 2024-09-17T23:27:11.118 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster2-mirror-image_18.status 2024-09-17T23:27:11.118 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster2-mirror-image_18.status 2024-09-17T23:27:11.118 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:27:11.118 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_18.status 2024-09-17T23:27:11.118 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster1-mirror-image_18.status 2024-09-17T23:27:11.118 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster1-mirror-image_18.status 2024-09-17T23:27:11.119 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:27:11.119 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_status_log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_18.status 2024-09-17T23:27:11.119 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ mirror_image_snapshot cluster2 mirror image_18 2024-09-17T23:27:11.119 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:27:11.119 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:27:11.119 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_18 2024-09-17T23:27:11.120 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_18 2024-09-17T23:27:12.183 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Snapshot ID: 120 2024-09-17T23:27:12.196 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_18 /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_18.status 2024-09-17T23:27:12.196 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:27:12.196 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:27:12.196 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_18 2024-09-17T23:27:12.197 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_18.status 2024-09-17T23:27:12.197 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_18 --format xml 2024-09-17T23:27:12.197 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:27:12.244 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-17T23:27:12.246 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snapshot_id=120 2024-09-17T23:27:12.246 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:27:12.246 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:27:12.247 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.2 2024-09-17T23:27:12.448 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_18.status 2024-09-17T23:27:12.448 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:27:12.448 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:27:12.448 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_18 2024-09-17T23:27:12.448 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_18.status 2024-09-17T23:27:12.448 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2024-09-17T23:27:12.448 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:27:12.500 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:27:12.500 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:27:12.501 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:27:12.501 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:27:12.501 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:27:12.502 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:27:12.502 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 120 = '' 2024-09-17T23:27:12.502 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:27:12.502 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.4 2024-09-17T23:27:12.903 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_18.status 2024-09-17T23:27:12.903 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:27:12.904 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:27:12.904 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_18 2024-09-17T23:27:12.904 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_18.status 2024-09-17T23:27:12.904 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2024-09-17T23:27:12.904 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:27:12.954 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:27:12.954 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:27:12.956 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:27:12.956 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:27:12.957 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:27:12.957 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:27:12.957 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 120 = '' 2024-09-17T23:27:12.957 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:27:12.957 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.8 2024-09-17T23:27:13.758 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_18.status 2024-09-17T23:27:13.758 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:27:13.758 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:27:13.758 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_18 2024-09-17T23:27:13.758 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_18.status 2024-09-17T23:27:13.759 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2024-09-17T23:27:13.759 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:27:13.806 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:27:13.808 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id=120 2024-09-17T23:27:13.808 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 120 = 120 2024-09-17T23:27:13.808 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:27:13.808 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snap_present cluster1 mirror image_18 snap 2024-09-17T23:27:13.808 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:27:13.808 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:27:13.808 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_18 2024-09-17T23:27:13.809 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:27:13.809 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:27:13.809 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-17T23:27:13.809 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:27:14.809 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 info mirror/image_18@snap 2024-09-17T23:27:14.858 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'image_18': 2024-09-17T23:27:14.859 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 128 MiB in 32 objects 2024-09-17T23:27:14.859 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:27:14.859 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 2 2024-09-17T23:27:14.859 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 10c69f4a6efc 2024-09-17T23:27:14.859 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.10c69f4a6efc 2024-09-17T23:27:14.859 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:27:14.859 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-17T23:27:14.859 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:27:14.859 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:27:14.859 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:19:19 2024 2024-09-17T23:27:14.859 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:19:19 2024 2024-09-17T23:27:14.859 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:19:19 2024 2024-09-17T23:27:14.860 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: protected: False 2024-09-17T23:27:14.860 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:27:14.860 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:27:14.860 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: 82669132-31fe-4405-bf71-3a31515c863e 2024-09-17T23:27:14.860 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: false 2024-09-17T23:27:14.863 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:27:14.863 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ compare_image_snaps mirror image_18 snap 2024-09-17T23:27:14.863 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:27:14.863 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_18 2024-09-17T23:27:14.863 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:27:14.863 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local ret=0 2024-09-17T23:27:14.863 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local rmt_export=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_18.export 2024-09-17T23:27:14.863 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local loc_export=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_18.export 2024-09-17T23:27:14.863 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_18.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_18.export 2024-09-17T23:27:14.864 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror export image_18@snap /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_18.export 2024-09-17T23:27:15.038 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-17T23:27:15.045 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 -p mirror export image_18@snap /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_18.export 2024-09-17T23:27:15.267 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-17T23:27:15.276 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ cmp /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_18.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_18.export 2024-09-17T23:27:15.339 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_18.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_18.export 2024-09-17T23:27:15.370 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:27:15.370 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:27:15.370 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_19 2024-09-17T23:27:15.371 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_snap cluster2 mirror image_19 snap 2024-09-17T23:27:15.371 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:27:15.371 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:27:15.371 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_19 2024-09-17T23:27:15.371 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:27:15.371 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror snap create image_19@snap --debug-rbd=20 --debug-journaler=20 2024-09-17T23:27:16.999 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_started cluster1 mirror image_19 2024-09-17T23:27:16.999 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:27:16.999 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:27:16.999 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_19 2024-09-17T23:27:16.999 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_state cluster1 mirror image_19 started 2024-09-17T23:27:16.999 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:27:16.999 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:27:17.000 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_19 2024-09-17T23:27:17.000 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=started 2024-09-17T23:27:17.000 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:27:17.000 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:27:17.000 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:27:18.001 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_replay_state cluster1 mirror image_19 started 2024-09-17T23:27:18.001 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:27:18.001 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:27:18.001 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_19 2024-09-17T23:27:18.001 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=started 2024-09-17T23:27:18.002 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_result 2024-09-17T23:27:18.002 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=stopped 2024-09-17T23:27:18.002 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_19 2024-09-17T23:27:18.002 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_instance=cluster1 2024-09-17T23:27:18.002 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:27:18.002 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ grep -i state 2024-09-17T23:27:18.002 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:27:18.002 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance=cluster1 2024-09-17T23:27:18.002 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local loop_instance 2024-09-17T23:27:18.002 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:27:18.002 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sleep 0 2024-09-17T23:27:18.005 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-17T23:27:18.005 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ seq 0 3 2024-09-17T23:27:18.005 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:27:18.005 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_19 2024-09-17T23:27:18.005 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:0 2024-09-17T23:27:18.005 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:27:18.006 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:27:18.006 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-17T23:27:18.006 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:0 2024-09-17T23:27:18.006 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:27:18.006 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:27:18.006 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:27:18.006 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=0 2024-09-17T23:27:18.006 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-17T23:27:18.006 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:27:18.006 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:27:18.006 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=0 2024-09-17T23:27:18.006 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:27:18.007 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-17T23:27:18.007 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:0 2024-09-17T23:27:18.007 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:27:18.007 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:27:18.007 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-17T23:27:18.007 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:0 2024-09-17T23:27:18.007 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:27:18.007 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:27:18.007 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:27:18.007 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:27:18.007 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-17T23:27:18.007 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:27:18.007 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:27:18.007 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=0 2024-09-17T23:27:18.007 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=0 2024-09-17T23:27:18.007 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-17T23:27:18.027 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:27:18.027 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:27:18.028 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:27:18.028 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_19 2024-09-17T23:27:18.174 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ return 0 2024-09-17T23:27:18.174 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ status_result=' "state": "Replaying"' 2024-09-17T23:27:18.174 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ echo ' "state": "Replaying"' 2024-09-17T23:27:18.174 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep -i Replaying 2024-09-17T23:27:18.175 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: "state": "Replaying" 2024-09-17T23:27:18.177 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_state=started 2024-09-17T23:27:18.177 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test started = started 2024-09-17T23:27:18.177 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:27:18.177 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_19 2024-09-17T23:27:18.177 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:27:18.177 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:27:18.177 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:27:18.177 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_19 2024-09-17T23:27:18.177 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = journal ']' 2024-09-17T23:27:18.177 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = snapshot ']' 2024-09-17T23:27:18.177 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_19 2024-09-17T23:27:18.177 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:27:18.177 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:27:18.178 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:27:18.178 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_19 2024-09-17T23:27:18.178 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster2-mirror-image_19.status 2024-09-17T23:27:18.178 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster2-mirror-image_19.status 2024-09-17T23:27:18.178 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:27:18.178 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_19.status 2024-09-17T23:27:18.179 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster1-mirror-image_19.status 2024-09-17T23:27:18.179 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster1-mirror-image_19.status 2024-09-17T23:27:18.179 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:27:18.179 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_status_log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_19.status 2024-09-17T23:27:18.179 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ mirror_image_snapshot cluster2 mirror image_19 2024-09-17T23:27:18.180 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:27:18.180 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:27:18.180 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_19 2024-09-17T23:27:18.180 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_19 2024-09-17T23:27:19.003 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Snapshot ID: 122 2024-09-17T23:27:19.016 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_19 /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_19.status 2024-09-17T23:27:19.016 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:27:19.016 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:27:19.016 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_19 2024-09-17T23:27:19.016 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_19.status 2024-09-17T23:27:19.017 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_19 --format xml 2024-09-17T23:27:19.017 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:27:19.065 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-17T23:27:19.067 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snapshot_id=122 2024-09-17T23:27:19.067 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:27:19.067 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:27:19.067 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.2 2024-09-17T23:27:19.268 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_19.status 2024-09-17T23:27:19.268 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:27:19.268 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:27:19.268 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_19 2024-09-17T23:27:19.269 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_19.status 2024-09-17T23:27:19.269 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2024-09-17T23:27:19.269 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:27:19.319 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:27:19.320 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:27:19.321 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:27:19.321 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:27:19.321 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:27:19.322 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:27:19.322 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 122 = '' 2024-09-17T23:27:19.322 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:27:19.322 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.4 2024-09-17T23:27:19.723 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_19.status 2024-09-17T23:27:19.723 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:27:19.723 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:27:19.723 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_19 2024-09-17T23:27:19.723 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_19.status 2024-09-17T23:27:19.723 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2024-09-17T23:27:19.723 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:27:19.776 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:27:19.776 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:27:19.777 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:27:19.778 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:27:19.778 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:27:19.778 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:27:19.778 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 122 = '' 2024-09-17T23:27:19.778 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:27:19.778 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.8 2024-09-17T23:27:20.580 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_19.status 2024-09-17T23:27:20.580 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:27:20.580 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:27:20.580 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_19 2024-09-17T23:27:20.580 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_19.status 2024-09-17T23:27:20.580 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2024-09-17T23:27:20.581 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:27:20.627 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:27:20.629 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id=122 2024-09-17T23:27:20.629 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 122 = 122 2024-09-17T23:27:20.629 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:27:20.629 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snap_present cluster1 mirror image_19 snap 2024-09-17T23:27:20.629 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:27:20.629 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:27:20.629 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_19 2024-09-17T23:27:20.629 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:27:20.629 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:27:20.629 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-17T23:27:20.629 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:27:21.630 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 info mirror/image_19@snap 2024-09-17T23:27:21.678 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'image_19': 2024-09-17T23:27:21.678 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 128 MiB in 32 objects 2024-09-17T23:27:21.678 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:27:21.678 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 2 2024-09-17T23:27:21.678 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 10c693ec657d 2024-09-17T23:27:21.678 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.10c693ec657d 2024-09-17T23:27:21.678 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:27:21.678 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-17T23:27:21.678 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:27:21.678 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:27:21.678 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:19:28 2024 2024-09-17T23:27:21.678 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:19:28 2024 2024-09-17T23:27:21.679 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:19:28 2024 2024-09-17T23:27:21.679 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: protected: False 2024-09-17T23:27:21.679 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:27:21.679 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:27:21.679 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: 263f2d9e-807d-4be1-b51d-f2bd7fb7a5d6 2024-09-17T23:27:21.679 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: false 2024-09-17T23:27:21.682 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:27:21.682 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ compare_image_snaps mirror image_19 snap 2024-09-17T23:27:21.682 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:27:21.682 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_19 2024-09-17T23:27:21.682 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:27:21.682 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local ret=0 2024-09-17T23:27:21.683 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local rmt_export=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_19.export 2024-09-17T23:27:21.683 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local loc_export=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_19.export 2024-09-17T23:27:21.683 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_19.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_19.export 2024-09-17T23:27:21.683 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror export image_19@snap /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_19.export 2024-09-17T23:27:21.860 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-17T23:27:21.869 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 -p mirror export image_19@snap /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_19.export 2024-09-17T23:27:22.109 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-17T23:27:22.118 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ cmp /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_19.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_19.export 2024-09-17T23:27:22.180 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_19.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_19.export 2024-09-17T23:27:22.213 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:27:22.213 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:27:22.213 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_20 2024-09-17T23:27:22.213 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_snap cluster2 mirror image_20 snap 2024-09-17T23:27:22.213 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:27:22.213 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:27:22.213 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_20 2024-09-17T23:27:22.213 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:27:22.213 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror snap create image_20@snap --debug-rbd=20 --debug-journaler=20 2024-09-17T23:27:24.021 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_started cluster1 mirror image_20 2024-09-17T23:27:24.021 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:27:24.021 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:27:24.021 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_20 2024-09-17T23:27:24.021 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_state cluster1 mirror image_20 started 2024-09-17T23:27:24.021 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:27:24.021 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:27:24.021 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_20 2024-09-17T23:27:24.021 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=started 2024-09-17T23:27:24.021 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:27:24.021 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:27:24.021 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:27:25.022 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_replay_state cluster1 mirror image_20 started 2024-09-17T23:27:25.023 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:27:25.023 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:27:25.023 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_20 2024-09-17T23:27:25.023 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=started 2024-09-17T23:27:25.023 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_result 2024-09-17T23:27:25.023 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=stopped 2024-09-17T23:27:25.023 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_20 2024-09-17T23:27:25.023 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_instance=cluster1 2024-09-17T23:27:25.023 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:27:25.023 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ grep -i state 2024-09-17T23:27:25.023 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:27:25.023 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance=cluster1 2024-09-17T23:27:25.024 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local loop_instance 2024-09-17T23:27:25.024 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:27:25.024 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sleep 0 2024-09-17T23:27:25.024 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-17T23:27:25.024 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ seq 0 3 2024-09-17T23:27:25.026 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:27:25.026 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_20 2024-09-17T23:27:25.026 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:0 2024-09-17T23:27:25.026 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:27:25.026 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:27:25.026 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-17T23:27:25.026 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:0 2024-09-17T23:27:25.026 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:27:25.026 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:27:25.026 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:27:25.026 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=0 2024-09-17T23:27:25.026 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-17T23:27:25.027 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:27:25.027 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:27:25.027 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=0 2024-09-17T23:27:25.027 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:27:25.027 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-17T23:27:25.027 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:0 2024-09-17T23:27:25.028 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:27:25.028 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:27:25.028 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-17T23:27:25.028 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:0 2024-09-17T23:27:25.028 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:27:25.028 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:27:25.028 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:27:25.029 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:27:25.029 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-17T23:27:25.029 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:27:25.029 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:27:25.029 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=0 2024-09-17T23:27:25.029 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=0 2024-09-17T23:27:25.029 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-17T23:27:25.050 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:27:25.050 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:27:25.050 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:27:25.051 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_20 2024-09-17T23:27:25.177 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ return 0 2024-09-17T23:27:25.178 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ status_result=' "state": "Replaying"' 2024-09-17T23:27:25.178 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ echo ' "state": "Replaying"' 2024-09-17T23:27:25.178 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep -i Replaying 2024-09-17T23:27:25.179 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: "state": "Replaying" 2024-09-17T23:27:25.180 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_state=started 2024-09-17T23:27:25.180 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test started = started 2024-09-17T23:27:25.181 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:27:25.181 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_20 2024-09-17T23:27:25.181 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:27:25.181 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:27:25.181 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:27:25.181 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_20 2024-09-17T23:27:25.181 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = journal ']' 2024-09-17T23:27:25.181 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = snapshot ']' 2024-09-17T23:27:25.181 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_20 2024-09-17T23:27:25.181 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:27:25.181 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:27:25.181 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:27:25.181 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_20 2024-09-17T23:27:25.181 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster2-mirror-image_20.status 2024-09-17T23:27:25.182 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster2-mirror-image_20.status 2024-09-17T23:27:25.182 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:27:25.182 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_20.status 2024-09-17T23:27:25.182 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster1-mirror-image_20.status 2024-09-17T23:27:25.182 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster1-mirror-image_20.status 2024-09-17T23:27:25.183 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:27:25.183 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_status_log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_20.status 2024-09-17T23:27:25.183 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ mirror_image_snapshot cluster2 mirror image_20 2024-09-17T23:27:25.183 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:27:25.183 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:27:25.184 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_20 2024-09-17T23:27:25.184 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_20 2024-09-17T23:27:26.033 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Snapshot ID: 124 2024-09-17T23:27:26.045 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_20 /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_20.status 2024-09-17T23:27:26.045 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:27:26.046 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:27:26.046 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_20 2024-09-17T23:27:26.046 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_20.status 2024-09-17T23:27:26.046 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_20 --format xml 2024-09-17T23:27:26.046 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:27:26.097 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-17T23:27:26.099 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snapshot_id=124 2024-09-17T23:27:26.099 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:27:26.099 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:27:26.099 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.2 2024-09-17T23:27:26.300 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_20.status 2024-09-17T23:27:26.301 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:27:26.301 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:27:26.301 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_20 2024-09-17T23:27:26.301 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_20.status 2024-09-17T23:27:26.301 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2024-09-17T23:27:26.301 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:27:26.353 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:27:26.353 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:27:26.355 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:27:26.355 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:27:26.355 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:27:26.355 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:27:26.355 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 124 = '' 2024-09-17T23:27:26.355 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:27:26.356 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.4 2024-09-17T23:27:26.757 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_20.status 2024-09-17T23:27:26.757 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:27:26.757 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:27:26.757 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_20 2024-09-17T23:27:26.757 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_20.status 2024-09-17T23:27:26.757 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2024-09-17T23:27:26.757 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:27:26.809 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:27:26.810 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:27:26.811 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:27:26.811 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:27:26.811 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:27:26.812 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:27:26.812 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 124 = '' 2024-09-17T23:27:26.812 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:27:26.812 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.8 2024-09-17T23:27:27.613 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_20.status 2024-09-17T23:27:27.614 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:27:27.614 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:27:27.614 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_20 2024-09-17T23:27:27.614 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_20.status 2024-09-17T23:27:27.614 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2024-09-17T23:27:27.614 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:27:27.661 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:27:27.663 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id=124 2024-09-17T23:27:27.663 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 124 = 124 2024-09-17T23:27:27.663 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:27:27.663 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snap_present cluster1 mirror image_20 snap 2024-09-17T23:27:27.663 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:27:27.663 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:27:27.664 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_20 2024-09-17T23:27:27.664 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:27:27.664 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:27:27.664 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-17T23:27:27.664 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:27:28.664 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 info mirror/image_20@snap 2024-09-17T23:27:28.712 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'image_20': 2024-09-17T23:27:28.712 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 128 MiB in 32 objects 2024-09-17T23:27:28.712 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:27:28.713 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 2 2024-09-17T23:27:28.713 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 10c65ad3bf53 2024-09-17T23:27:28.713 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.10c65ad3bf53 2024-09-17T23:27:28.713 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:27:28.713 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-17T23:27:28.713 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:27:28.713 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:27:28.713 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:19:43 2024 2024-09-17T23:27:28.713 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:19:43 2024 2024-09-17T23:27:28.714 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:19:43 2024 2024-09-17T23:27:28.714 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: protected: False 2024-09-17T23:27:28.714 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:27:28.714 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:27:28.714 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: 04cb45cc-a535-47bd-b9a9-46bf5ca4fe67 2024-09-17T23:27:28.714 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: false 2024-09-17T23:27:28.716 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:27:28.716 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ compare_image_snaps mirror image_20 snap 2024-09-17T23:27:28.716 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:27:28.716 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_20 2024-09-17T23:27:28.716 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:27:28.716 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local ret=0 2024-09-17T23:27:28.716 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local rmt_export=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_20.export 2024-09-17T23:27:28.716 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local loc_export=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_20.export 2024-09-17T23:27:28.717 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_20.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_20.export 2024-09-17T23:27:28.717 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror export image_20@snap /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_20.export 2024-09-17T23:27:28.882 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-17T23:27:28.889 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 -p mirror export image_20@snap /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_20.export 2024-09-17T23:27:29.091 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-17T23:27:29.099 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ cmp /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_20.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_20.export 2024-09-17T23:27:29.161 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_20.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_20.export 2024-09-17T23:27:29.193 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:27:29.193 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:27:29.193 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_21 2024-09-17T23:27:29.193 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_snap cluster2 mirror image_21 snap 2024-09-17T23:27:29.193 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:27:29.194 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:27:29.194 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_21 2024-09-17T23:27:29.194 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:27:29.194 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror snap create image_21@snap --debug-rbd=20 --debug-journaler=20 2024-09-17T23:27:31.067 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_started cluster1 mirror image_21 2024-09-17T23:27:31.067 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:27:31.067 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:27:31.067 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_21 2024-09-17T23:27:31.067 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_state cluster1 mirror image_21 started 2024-09-17T23:27:31.067 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:27:31.067 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:27:31.067 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_21 2024-09-17T23:27:31.068 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=started 2024-09-17T23:27:31.068 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:27:31.068 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:27:31.068 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:27:32.068 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_replay_state cluster1 mirror image_21 started 2024-09-17T23:27:32.069 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:27:32.069 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:27:32.069 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_21 2024-09-17T23:27:32.069 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=started 2024-09-17T23:27:32.069 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_result 2024-09-17T23:27:32.069 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=stopped 2024-09-17T23:27:32.070 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_21 2024-09-17T23:27:32.070 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_instance=cluster1 2024-09-17T23:27:32.070 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:27:32.070 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ grep -i state 2024-09-17T23:27:32.070 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:27:32.070 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance=cluster1 2024-09-17T23:27:32.070 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local loop_instance 2024-09-17T23:27:32.070 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:27:32.070 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sleep 0 2024-09-17T23:27:32.071 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-17T23:27:32.072 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ seq 0 3 2024-09-17T23:27:32.072 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:27:32.072 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_21 2024-09-17T23:27:32.073 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:0 2024-09-17T23:27:32.073 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:27:32.073 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:27:32.073 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-17T23:27:32.073 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:0 2024-09-17T23:27:32.073 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:27:32.073 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:27:32.073 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:27:32.073 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=0 2024-09-17T23:27:32.073 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-17T23:27:32.073 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:27:32.073 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:27:32.073 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=0 2024-09-17T23:27:32.074 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:27:32.074 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-17T23:27:32.074 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:0 2024-09-17T23:27:32.074 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:27:32.074 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:27:32.074 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-17T23:27:32.075 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:0 2024-09-17T23:27:32.075 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:27:32.075 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:27:32.075 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:27:32.075 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:27:32.075 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-17T23:27:32.075 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:27:32.076 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:27:32.076 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=0 2024-09-17T23:27:32.076 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=0 2024-09-17T23:27:32.076 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-17T23:27:32.096 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:27:32.096 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:27:32.096 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:27:32.096 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_21 2024-09-17T23:27:32.222 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ return 0 2024-09-17T23:27:32.223 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ status_result=' "state": "Replaying"' 2024-09-17T23:27:32.223 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ echo ' "state": "Replaying"' 2024-09-17T23:27:32.223 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep -i Replaying 2024-09-17T23:27:32.224 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: "state": "Replaying" 2024-09-17T23:27:32.225 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_state=started 2024-09-17T23:27:32.225 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test started = started 2024-09-17T23:27:32.225 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:27:32.225 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_21 2024-09-17T23:27:32.225 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:27:32.225 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:27:32.225 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:27:32.225 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_21 2024-09-17T23:27:32.226 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = journal ']' 2024-09-17T23:27:32.226 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = snapshot ']' 2024-09-17T23:27:32.226 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_21 2024-09-17T23:27:32.226 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:27:32.226 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:27:32.226 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:27:32.226 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_21 2024-09-17T23:27:32.226 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster2-mirror-image_21.status 2024-09-17T23:27:32.226 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster2-mirror-image_21.status 2024-09-17T23:27:32.226 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:27:32.227 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_21.status 2024-09-17T23:27:32.227 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster1-mirror-image_21.status 2024-09-17T23:27:32.227 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster1-mirror-image_21.status 2024-09-17T23:27:32.227 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:27:32.228 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_status_log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_21.status 2024-09-17T23:27:32.228 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ mirror_image_snapshot cluster2 mirror image_21 2024-09-17T23:27:32.228 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:27:32.228 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:27:32.228 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_21 2024-09-17T23:27:32.228 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_21 2024-09-17T23:27:33.071 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Snapshot ID: 126 2024-09-17T23:27:33.083 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_21 /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_21.status 2024-09-17T23:27:33.084 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:27:33.084 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:27:33.084 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_21 2024-09-17T23:27:33.084 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_21.status 2024-09-17T23:27:33.084 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_21 --format xml 2024-09-17T23:27:33.084 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:27:33.132 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-17T23:27:33.134 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snapshot_id=126 2024-09-17T23:27:33.134 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:27:33.134 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:27:33.134 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.2 2024-09-17T23:27:33.335 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_21.status 2024-09-17T23:27:33.336 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:27:33.336 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:27:33.336 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_21 2024-09-17T23:27:33.336 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_21.status 2024-09-17T23:27:33.336 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2024-09-17T23:27:33.336 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:27:33.386 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:27:33.387 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:27:33.388 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:27:33.388 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:27:33.388 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:27:33.389 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:27:33.389 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 126 = '' 2024-09-17T23:27:33.389 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:27:33.389 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.4 2024-09-17T23:27:33.790 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_21.status 2024-09-17T23:27:33.791 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:27:33.791 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:27:33.791 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_21 2024-09-17T23:27:33.791 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_21.status 2024-09-17T23:27:33.791 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2024-09-17T23:27:33.791 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:27:33.843 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:27:33.844 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:27:33.845 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:27:33.845 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:27:33.845 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:27:33.846 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:27:33.846 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 126 = '' 2024-09-17T23:27:33.846 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:27:33.846 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.8 2024-09-17T23:27:34.647 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_21.status 2024-09-17T23:27:34.648 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:27:34.648 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:27:34.648 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_21 2024-09-17T23:27:34.648 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_21.status 2024-09-17T23:27:34.648 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2024-09-17T23:27:34.648 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:27:34.694 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:27:34.696 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id=126 2024-09-17T23:27:34.696 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 126 = 126 2024-09-17T23:27:34.696 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:27:34.696 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snap_present cluster1 mirror image_21 snap 2024-09-17T23:27:34.696 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:27:34.696 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:27:34.696 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_21 2024-09-17T23:27:34.696 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:27:34.696 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:27:34.696 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-17T23:27:34.697 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:27:35.697 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 info mirror/image_21@snap 2024-09-17T23:27:35.744 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'image_21': 2024-09-17T23:27:35.744 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 128 MiB in 32 objects 2024-09-17T23:27:35.744 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:27:35.744 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 2 2024-09-17T23:27:35.744 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 10c6e95c8f0d 2024-09-17T23:27:35.744 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.10c6e95c8f0d 2024-09-17T23:27:35.744 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:27:35.745 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-17T23:27:35.745 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:27:35.745 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:27:35.745 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:19:56 2024 2024-09-17T23:27:35.745 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:19:56 2024 2024-09-17T23:27:35.745 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:19:56 2024 2024-09-17T23:27:35.745 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: protected: False 2024-09-17T23:27:35.745 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:27:35.745 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:27:35.745 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: 7f670507-4ecc-426a-a941-5287c614d52d 2024-09-17T23:27:35.745 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: false 2024-09-17T23:27:35.748 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:27:35.748 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ compare_image_snaps mirror image_21 snap 2024-09-17T23:27:35.748 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:27:35.748 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_21 2024-09-17T23:27:35.749 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:27:35.749 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local ret=0 2024-09-17T23:27:35.749 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local rmt_export=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_21.export 2024-09-17T23:27:35.749 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local loc_export=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_21.export 2024-09-17T23:27:35.749 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_21.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_21.export 2024-09-17T23:27:35.749 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror export image_21@snap /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_21.export 2024-09-17T23:27:35.924 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-17T23:27:35.931 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 -p mirror export image_21@snap /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_21.export 2024-09-17T23:27:36.188 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-17T23:27:36.195 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ cmp /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_21.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_21.export 2024-09-17T23:27:36.257 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_21.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_21.export 2024-09-17T23:27:36.289 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:27:36.289 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:27:36.289 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_22 2024-09-17T23:27:36.289 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_snap cluster2 mirror image_22 snap 2024-09-17T23:27:36.289 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:27:36.289 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:27:36.289 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_22 2024-09-17T23:27:36.289 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:27:36.289 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror snap create image_22@snap --debug-rbd=20 --debug-journaler=20 2024-09-17T23:27:38.089 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_started cluster1 mirror image_22 2024-09-17T23:27:38.089 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:27:38.090 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:27:38.090 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_22 2024-09-17T23:27:38.090 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_state cluster1 mirror image_22 started 2024-09-17T23:27:38.090 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:27:38.090 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:27:38.090 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_22 2024-09-17T23:27:38.090 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=started 2024-09-17T23:27:38.090 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:27:38.090 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:27:38.090 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:27:39.090 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_replay_state cluster1 mirror image_22 started 2024-09-17T23:27:39.091 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:27:39.091 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:27:39.091 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_22 2024-09-17T23:27:39.091 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=started 2024-09-17T23:27:39.091 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_result 2024-09-17T23:27:39.091 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=stopped 2024-09-17T23:27:39.092 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_22 2024-09-17T23:27:39.092 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_instance=cluster1 2024-09-17T23:27:39.092 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ grep -i state 2024-09-17T23:27:39.092 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:27:39.092 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:27:39.092 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance=cluster1 2024-09-17T23:27:39.092 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local loop_instance 2024-09-17T23:27:39.092 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:27:39.093 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sleep 0 2024-09-17T23:27:39.093 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-17T23:27:39.093 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ seq 0 3 2024-09-17T23:27:39.095 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:27:39.095 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_22 2024-09-17T23:27:39.095 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:0 2024-09-17T23:27:39.095 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:27:39.095 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:27:39.095 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-17T23:27:39.095 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:0 2024-09-17T23:27:39.095 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:27:39.096 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:27:39.096 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:27:39.096 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=0 2024-09-17T23:27:39.096 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-17T23:27:39.096 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:27:39.096 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:27:39.096 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=0 2024-09-17T23:27:39.096 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:27:39.096 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-17T23:27:39.096 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:0 2024-09-17T23:27:39.096 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:27:39.096 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:27:39.096 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-17T23:27:39.096 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:0 2024-09-17T23:27:39.096 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:27:39.096 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:27:39.097 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:27:39.097 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:27:39.097 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-17T23:27:39.098 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:27:39.098 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:27:39.098 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=0 2024-09-17T23:27:39.098 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=0 2024-09-17T23:27:39.098 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-17T23:27:39.118 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:27:39.118 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:27:39.118 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:27:39.118 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_22 2024-09-17T23:27:39.245 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ return 0 2024-09-17T23:27:39.245 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ status_result=' "state": "Replaying"' 2024-09-17T23:27:39.245 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ echo ' "state": "Replaying"' 2024-09-17T23:27:39.246 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep -i Replaying 2024-09-17T23:27:39.246 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: "state": "Replaying" 2024-09-17T23:27:39.248 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_state=started 2024-09-17T23:27:39.248 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test started = started 2024-09-17T23:27:39.248 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:27:39.248 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_22 2024-09-17T23:27:39.248 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:27:39.248 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:27:39.248 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:27:39.248 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_22 2024-09-17T23:27:39.248 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = journal ']' 2024-09-17T23:27:39.248 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = snapshot ']' 2024-09-17T23:27:39.248 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_22 2024-09-17T23:27:39.248 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:27:39.248 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:27:39.249 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:27:39.249 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_22 2024-09-17T23:27:39.249 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster2-mirror-image_22.status 2024-09-17T23:27:39.249 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster2-mirror-image_22.status 2024-09-17T23:27:39.249 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:27:39.249 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_22.status 2024-09-17T23:27:39.250 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster1-mirror-image_22.status 2024-09-17T23:27:39.250 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster1-mirror-image_22.status 2024-09-17T23:27:39.250 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:27:39.250 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_status_log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_22.status 2024-09-17T23:27:39.250 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ mirror_image_snapshot cluster2 mirror image_22 2024-09-17T23:27:39.251 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:27:39.251 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:27:39.251 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_22 2024-09-17T23:27:39.251 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_22 2024-09-17T23:27:40.100 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Snapshot ID: 128 2024-09-17T23:27:40.112 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_22 /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_22.status 2024-09-17T23:27:40.113 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:27:40.113 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:27:40.113 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_22 2024-09-17T23:27:40.113 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_22.status 2024-09-17T23:27:40.113 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_22 --format xml 2024-09-17T23:27:40.113 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:27:40.161 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-17T23:27:40.163 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snapshot_id=128 2024-09-17T23:27:40.163 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:27:40.163 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:27:40.163 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.2 2024-09-17T23:27:40.364 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_22.status 2024-09-17T23:27:40.365 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:27:40.365 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:27:40.365 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_22 2024-09-17T23:27:40.365 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_22.status 2024-09-17T23:27:40.365 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2024-09-17T23:27:40.365 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:27:40.425 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:27:40.425 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:27:40.427 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:27:40.427 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:27:40.427 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:27:40.427 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:27:40.427 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 128 = '' 2024-09-17T23:27:40.427 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:27:40.427 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.4 2024-09-17T23:27:40.829 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_22.status 2024-09-17T23:27:40.829 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:27:40.829 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:27:40.829 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_22 2024-09-17T23:27:40.829 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_22.status 2024-09-17T23:27:40.829 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2024-09-17T23:27:40.829 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:27:40.883 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:27:40.883 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:27:40.885 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:27:40.885 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:27:40.885 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:27:40.885 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:27:40.886 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 128 = '' 2024-09-17T23:27:40.886 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:27:40.886 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.8 2024-09-17T23:27:41.687 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_22.status 2024-09-17T23:27:41.687 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:27:41.687 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:27:41.688 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_22 2024-09-17T23:27:41.688 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_22.status 2024-09-17T23:27:41.688 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2024-09-17T23:27:41.688 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:27:41.739 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:27:41.741 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id=128 2024-09-17T23:27:41.741 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 128 = 128 2024-09-17T23:27:41.741 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:27:41.741 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snap_present cluster1 mirror image_22 snap 2024-09-17T23:27:41.741 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:27:41.741 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:27:41.741 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_22 2024-09-17T23:27:41.742 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:27:41.742 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:27:41.742 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-17T23:27:41.742 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:27:42.742 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 info mirror/image_22@snap 2024-09-17T23:27:42.791 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'image_22': 2024-09-17T23:27:42.791 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 128 MiB in 32 objects 2024-09-17T23:27:42.791 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:27:42.792 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 2 2024-09-17T23:27:42.792 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 10c613eb8516 2024-09-17T23:27:42.792 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.10c613eb8516 2024-09-17T23:27:42.792 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:27:42.792 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-17T23:27:42.792 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:27:42.792 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:27:42.792 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:20:07 2024 2024-09-17T23:27:42.792 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:20:07 2024 2024-09-17T23:27:42.792 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:20:07 2024 2024-09-17T23:27:42.792 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: protected: False 2024-09-17T23:27:42.792 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:27:42.792 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:27:42.792 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: 1068bb88-a6ef-49c0-a048-fafa200170dc 2024-09-17T23:27:42.793 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: false 2024-09-17T23:27:42.796 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:27:42.796 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ compare_image_snaps mirror image_22 snap 2024-09-17T23:27:42.797 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:27:42.797 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_22 2024-09-17T23:27:42.797 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:27:42.797 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local ret=0 2024-09-17T23:27:42.797 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local rmt_export=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_22.export 2024-09-17T23:27:42.797 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local loc_export=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_22.export 2024-09-17T23:27:42.798 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_22.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_22.export 2024-09-17T23:27:42.798 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror export image_22@snap /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_22.export 2024-09-17T23:27:42.962 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-17T23:27:42.970 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 -p mirror export image_22@snap /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_22.export 2024-09-17T23:27:43.186 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-17T23:27:43.193 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ cmp /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_22.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_22.export 2024-09-17T23:27:43.256 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_22.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_22.export 2024-09-17T23:27:43.287 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:27:43.287 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:27:43.287 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_23 2024-09-17T23:27:43.288 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_snap cluster2 mirror image_23 snap 2024-09-17T23:27:43.288 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:27:43.288 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:27:43.288 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_23 2024-09-17T23:27:43.288 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:27:43.288 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror snap create image_23@snap --debug-rbd=20 --debug-journaler=20 2024-09-17T23:27:45.126 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_started cluster1 mirror image_23 2024-09-17T23:27:45.126 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:27:45.126 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:27:45.126 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_23 2024-09-17T23:27:45.127 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_state cluster1 mirror image_23 started 2024-09-17T23:27:45.127 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:27:45.127 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:27:45.127 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_23 2024-09-17T23:27:45.127 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=started 2024-09-17T23:27:45.127 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:27:45.127 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:27:45.127 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:27:46.127 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_replay_state cluster1 mirror image_23 started 2024-09-17T23:27:46.128 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:27:46.128 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:27:46.128 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_23 2024-09-17T23:27:46.128 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=started 2024-09-17T23:27:46.128 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_result 2024-09-17T23:27:46.128 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=stopped 2024-09-17T23:27:46.129 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_23 2024-09-17T23:27:46.129 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ grep -i state 2024-09-17T23:27:46.129 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_instance=cluster1 2024-09-17T23:27:46.129 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:27:46.129 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:27:46.129 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance=cluster1 2024-09-17T23:27:46.129 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local loop_instance 2024-09-17T23:27:46.129 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:27:46.129 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sleep 0 2024-09-17T23:27:46.130 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-17T23:27:46.130 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ seq 0 3 2024-09-17T23:27:46.132 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:27:46.132 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_23 2024-09-17T23:27:46.132 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:0 2024-09-17T23:27:46.132 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:27:46.132 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:27:46.133 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-17T23:27:46.133 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:0 2024-09-17T23:27:46.133 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:27:46.133 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:27:46.133 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:27:46.133 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=0 2024-09-17T23:27:46.133 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-17T23:27:46.133 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:27:46.133 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:27:46.133 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=0 2024-09-17T23:27:46.133 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:27:46.133 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-17T23:27:46.133 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:0 2024-09-17T23:27:46.133 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:27:46.133 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:27:46.134 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-17T23:27:46.134 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:0 2024-09-17T23:27:46.134 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:27:46.134 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:27:46.134 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:27:46.134 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:27:46.134 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-17T23:27:46.134 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:27:46.134 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:27:46.134 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=0 2024-09-17T23:27:46.134 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=0 2024-09-17T23:27:46.134 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-17T23:27:46.154 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:27:46.155 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:27:46.155 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:27:46.155 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_23 2024-09-17T23:27:46.282 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ return 0 2024-09-17T23:27:46.283 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ status_result=' "state": "Replaying"' 2024-09-17T23:27:46.283 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ echo ' "state": "Replaying"' 2024-09-17T23:27:46.283 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep -i Replaying 2024-09-17T23:27:46.284 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: "state": "Replaying" 2024-09-17T23:27:46.285 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_state=started 2024-09-17T23:27:46.285 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test started = started 2024-09-17T23:27:46.285 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:27:46.285 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_23 2024-09-17T23:27:46.285 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:27:46.285 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:27:46.286 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:27:46.286 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_23 2024-09-17T23:27:46.286 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = journal ']' 2024-09-17T23:27:46.286 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = snapshot ']' 2024-09-17T23:27:46.286 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_23 2024-09-17T23:27:46.286 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:27:46.286 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:27:46.286 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:27:46.286 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_23 2024-09-17T23:27:46.286 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster2-mirror-image_23.status 2024-09-17T23:27:46.286 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster2-mirror-image_23.status 2024-09-17T23:27:46.286 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:27:46.287 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_23.status 2024-09-17T23:27:46.287 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster1-mirror-image_23.status 2024-09-17T23:27:46.287 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster1-mirror-image_23.status 2024-09-17T23:27:46.287 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:27:46.288 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_status_log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_23.status 2024-09-17T23:27:46.288 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ mirror_image_snapshot cluster2 mirror image_23 2024-09-17T23:27:46.288 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:27:46.288 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:27:46.288 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_23 2024-09-17T23:27:46.288 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_23 2024-09-17T23:27:47.137 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Snapshot ID: 130 2024-09-17T23:27:47.149 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_23 /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_23.status 2024-09-17T23:27:47.150 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:27:47.150 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:27:47.150 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_23 2024-09-17T23:27:47.150 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_23.status 2024-09-17T23:27:47.150 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_23 --format xml 2024-09-17T23:27:47.150 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:27:47.199 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-17T23:27:47.201 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snapshot_id=130 2024-09-17T23:27:47.201 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:27:47.201 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:27:47.201 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.2 2024-09-17T23:27:47.402 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_23.status 2024-09-17T23:27:47.402 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:27:47.402 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:27:47.403 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_23 2024-09-17T23:27:47.403 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_23.status 2024-09-17T23:27:47.403 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2024-09-17T23:27:47.403 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:27:47.453 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:27:47.453 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:27:47.454 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:27:47.455 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:27:47.455 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:27:47.456 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:27:47.456 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 130 = '' 2024-09-17T23:27:47.456 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:27:47.456 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.4 2024-09-17T23:27:47.857 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_23.status 2024-09-17T23:27:47.857 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:27:47.857 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:27:47.857 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_23 2024-09-17T23:27:47.857 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_23.status 2024-09-17T23:27:47.857 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2024-09-17T23:27:47.857 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:27:47.910 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:27:47.910 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:27:47.912 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:27:47.913 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:27:47.913 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:27:47.913 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:27:47.913 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 130 = '' 2024-09-17T23:27:47.913 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:27:47.913 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.8 2024-09-17T23:27:48.714 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_23.status 2024-09-17T23:27:48.714 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:27:48.714 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:27:48.714 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_23 2024-09-17T23:27:48.715 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_23.status 2024-09-17T23:27:48.715 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2024-09-17T23:27:48.715 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:27:48.763 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:27:48.765 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id=130 2024-09-17T23:27:48.765 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 130 = 130 2024-09-17T23:27:48.765 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:27:48.765 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snap_present cluster1 mirror image_23 snap 2024-09-17T23:27:48.765 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:27:48.765 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:27:48.765 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_23 2024-09-17T23:27:48.766 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:27:48.766 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:27:48.766 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-17T23:27:48.766 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:27:49.766 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 info mirror/image_23@snap 2024-09-17T23:27:49.814 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'image_23': 2024-09-17T23:27:49.814 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 128 MiB in 32 objects 2024-09-17T23:27:49.814 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:27:49.814 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 2 2024-09-17T23:27:49.814 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 10c6f023526f 2024-09-17T23:27:49.814 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.10c6f023526f 2024-09-17T23:27:49.814 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:27:49.814 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-17T23:27:49.814 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:27:49.814 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:27:49.815 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:20:13 2024 2024-09-17T23:27:49.815 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:20:13 2024 2024-09-17T23:27:49.815 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:20:13 2024 2024-09-17T23:27:49.815 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: protected: False 2024-09-17T23:27:49.815 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:27:49.815 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:27:49.815 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: 7b178615-290c-4f56-9926-be968e7bf83a 2024-09-17T23:27:49.815 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: false 2024-09-17T23:27:49.817 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:27:49.818 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ compare_image_snaps mirror image_23 snap 2024-09-17T23:27:49.818 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:27:49.818 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_23 2024-09-17T23:27:49.818 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:27:49.818 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local ret=0 2024-09-17T23:27:49.818 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local rmt_export=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_23.export 2024-09-17T23:27:49.818 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local loc_export=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_23.export 2024-09-17T23:27:49.818 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_23.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_23.export 2024-09-17T23:27:49.818 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror export image_23@snap /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_23.export 2024-09-17T23:27:49.987 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-17T23:27:49.993 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 -p mirror export image_23@snap /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_23.export 2024-09-17T23:27:50.213 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-17T23:27:50.220 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ cmp /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_23.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_23.export 2024-09-17T23:27:50.283 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_23.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_23.export 2024-09-17T23:27:50.314 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:27:50.314 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:27:50.314 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_24 2024-09-17T23:27:50.315 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_snap cluster2 mirror image_24 snap 2024-09-17T23:27:50.315 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:27:50.315 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:27:50.315 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_24 2024-09-17T23:27:50.315 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:27:50.315 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror snap create image_24@snap --debug-rbd=20 --debug-journaler=20 2024-09-17T23:27:52.161 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_started cluster1 mirror image_24 2024-09-17T23:27:52.162 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:27:52.162 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:27:52.162 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_24 2024-09-17T23:27:52.162 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_state cluster1 mirror image_24 started 2024-09-17T23:27:52.162 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:27:52.162 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:27:52.162 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_24 2024-09-17T23:27:52.162 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=started 2024-09-17T23:27:52.162 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:27:52.162 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:27:52.162 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:27:52.846 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.5686654845065625) > weight (0.25) 2024-09-17T23:27:52.846 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.9399182424222274) > weight (0.25) 2024-09-17T23:27:52.846 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.7274610082183383) > weight (0.25) 2024-09-17T23:27:52.846 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.4807001901790938) > weight (0.25) 2024-09-17T23:27:52.846 INFO:tasks.rbd_mirror_thrash:waiting for 109 secs before thrashing 2024-09-17T23:27:53.163 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_replay_state cluster1 mirror image_24 started 2024-09-17T23:27:53.163 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:27:53.163 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:27:53.163 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_24 2024-09-17T23:27:53.163 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=started 2024-09-17T23:27:53.163 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_result 2024-09-17T23:27:53.163 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=stopped 2024-09-17T23:27:53.164 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_24 2024-09-17T23:27:53.164 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_instance=cluster1 2024-09-17T23:27:53.165 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:27:53.165 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ grep -i state 2024-09-17T23:27:53.165 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:27:53.165 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance=cluster1 2024-09-17T23:27:53.165 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local loop_instance 2024-09-17T23:27:53.165 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:27:53.165 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sleep 0 2024-09-17T23:27:53.165 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-17T23:27:53.166 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ seq 0 3 2024-09-17T23:27:53.167 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:27:53.167 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_24 2024-09-17T23:27:53.167 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:0 2024-09-17T23:27:53.168 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:27:53.168 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:27:53.168 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-17T23:27:53.168 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:0 2024-09-17T23:27:53.168 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:27:53.168 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:27:53.168 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:27:53.168 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=0 2024-09-17T23:27:53.168 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-17T23:27:53.168 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:27:53.168 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:27:53.168 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=0 2024-09-17T23:27:53.168 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:27:53.168 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-17T23:27:53.169 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:0 2024-09-17T23:27:53.169 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:27:53.169 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:27:53.169 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-17T23:27:53.169 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:0 2024-09-17T23:27:53.169 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:27:53.169 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:27:53.169 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:27:53.169 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:27:53.170 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-17T23:27:53.170 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:27:53.170 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:27:53.170 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=0 2024-09-17T23:27:53.170 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=0 2024-09-17T23:27:53.170 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-17T23:27:53.191 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:27:53.191 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:27:53.191 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:27:53.191 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_24 2024-09-17T23:27:53.317 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ return 0 2024-09-17T23:27:53.318 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ status_result=' "state": "Replaying"' 2024-09-17T23:27:53.318 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ echo ' "state": "Replaying"' 2024-09-17T23:27:53.318 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep -i Replaying 2024-09-17T23:27:53.319 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: "state": "Replaying" 2024-09-17T23:27:53.320 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_state=started 2024-09-17T23:27:53.320 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test started = started 2024-09-17T23:27:53.320 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:27:53.320 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_24 2024-09-17T23:27:53.320 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:27:53.320 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:27:53.320 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:27:53.321 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_24 2024-09-17T23:27:53.321 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = journal ']' 2024-09-17T23:27:53.321 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = snapshot ']' 2024-09-17T23:27:53.321 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_24 2024-09-17T23:27:53.321 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:27:53.321 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:27:53.321 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:27:53.321 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_24 2024-09-17T23:27:53.321 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster2-mirror-image_24.status 2024-09-17T23:27:53.321 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster2-mirror-image_24.status 2024-09-17T23:27:53.321 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:27:53.322 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_24.status 2024-09-17T23:27:53.322 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster1-mirror-image_24.status 2024-09-17T23:27:53.322 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster1-mirror-image_24.status 2024-09-17T23:27:53.322 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:27:53.323 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_status_log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_24.status 2024-09-17T23:27:53.323 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ mirror_image_snapshot cluster2 mirror image_24 2024-09-17T23:27:53.323 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:27:53.323 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:27:53.323 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_24 2024-09-17T23:27:53.323 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_24 2024-09-17T23:27:54.162 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Snapshot ID: 132 2024-09-17T23:27:54.174 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_24 /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_24.status 2024-09-17T23:27:54.174 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:27:54.174 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:27:54.174 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_24 2024-09-17T23:27:54.174 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_24.status 2024-09-17T23:27:54.175 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_24 --format xml 2024-09-17T23:27:54.175 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:27:54.222 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-17T23:27:54.224 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snapshot_id=132 2024-09-17T23:27:54.224 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:27:54.224 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:27:54.224 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.2 2024-09-17T23:27:54.426 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_24.status 2024-09-17T23:27:54.426 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:27:54.426 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:27:54.426 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_24 2024-09-17T23:27:54.426 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_24.status 2024-09-17T23:27:54.426 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2024-09-17T23:27:54.426 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:27:54.477 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:27:54.477 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:27:54.479 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:27:54.479 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:27:54.479 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:27:54.480 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:27:54.480 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 132 = '' 2024-09-17T23:27:54.480 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:27:54.480 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.4 2024-09-17T23:27:54.881 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_24.status 2024-09-17T23:27:54.881 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:27:54.881 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:27:54.881 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_24 2024-09-17T23:27:54.881 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_24.status 2024-09-17T23:27:54.881 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2024-09-17T23:27:54.881 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:27:54.934 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:27:54.934 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:27:54.935 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:27:54.935 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:27:54.936 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:27:54.936 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:27:54.936 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 132 = '' 2024-09-17T23:27:54.936 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:27:54.936 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.8 2024-09-17T23:27:55.738 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_24.status 2024-09-17T23:27:55.738 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:27:55.738 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:27:55.738 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_24 2024-09-17T23:27:55.738 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_24.status 2024-09-17T23:27:55.738 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2024-09-17T23:27:55.738 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:27:55.786 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:27:55.789 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id=132 2024-09-17T23:27:55.789 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 132 = 132 2024-09-17T23:27:55.789 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:27:55.789 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snap_present cluster1 mirror image_24 snap 2024-09-17T23:27:55.789 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:27:55.790 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:27:55.790 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_24 2024-09-17T23:27:55.790 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:27:55.790 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:27:55.790 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-17T23:27:55.790 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:27:56.789 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 info mirror/image_24@snap 2024-09-17T23:27:56.839 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'image_24': 2024-09-17T23:27:56.839 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 128 MiB in 32 objects 2024-09-17T23:27:56.839 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:27:56.839 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 2 2024-09-17T23:27:56.839 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 10c6608dd479 2024-09-17T23:27:56.839 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.10c6608dd479 2024-09-17T23:27:56.839 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:27:56.839 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-17T23:27:56.839 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:27:56.839 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:27:56.839 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:20:27 2024 2024-09-17T23:27:56.839 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:20:27 2024 2024-09-17T23:27:56.840 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:20:27 2024 2024-09-17T23:27:56.840 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: protected: False 2024-09-17T23:27:56.840 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:27:56.840 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:27:56.840 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: de674b7e-029c-4272-9b26-d163572b5aea 2024-09-17T23:27:56.840 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: false 2024-09-17T23:27:56.842 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:27:56.843 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ compare_image_snaps mirror image_24 snap 2024-09-17T23:27:56.843 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:27:56.843 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_24 2024-09-17T23:27:56.843 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:27:56.843 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local ret=0 2024-09-17T23:27:56.843 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local rmt_export=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_24.export 2024-09-17T23:27:56.843 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local loc_export=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_24.export 2024-09-17T23:27:56.843 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_24.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_24.export 2024-09-17T23:27:56.843 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror export image_24@snap /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_24.export 2024-09-17T23:27:57.037 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-17T23:27:57.045 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 -p mirror export image_24@snap /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_24.export 2024-09-17T23:27:57.413 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-17T23:27:57.421 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ cmp /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_24.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_24.export 2024-09-17T23:27:57.483 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_24.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_24.export 2024-09-17T23:27:57.515 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:27:57.516 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:27:57.516 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_25 2024-09-17T23:27:57.516 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_snap cluster2 mirror image_25 snap 2024-09-17T23:27:57.516 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:27:57.516 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:27:57.516 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_25 2024-09-17T23:27:57.516 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:27:57.516 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror snap create image_25@snap --debug-rbd=20 --debug-journaler=20 2024-09-17T23:27:57.954 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.5894508368362236) > weight (0.25) 2024-09-17T23:27:57.954 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.3666328570144054) > weight (0.25) 2024-09-17T23:27:57.954 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.28044204732635425) > weight (0.25) 2024-09-17T23:27:57.954 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.39082339751982675) > weight (0.25) 2024-09-17T23:27:57.954 INFO:tasks.rbd_mirror_thrash:waiting for 67 secs before thrashing 2024-09-17T23:27:59.192 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_started cluster1 mirror image_25 2024-09-17T23:27:59.192 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:27:59.192 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:27:59.192 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_25 2024-09-17T23:27:59.193 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_state cluster1 mirror image_25 started 2024-09-17T23:27:59.193 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:27:59.193 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:27:59.193 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_25 2024-09-17T23:27:59.193 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=started 2024-09-17T23:27:59.193 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:27:59.193 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:27:59.193 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:28:00.193 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_replay_state cluster1 mirror image_25 started 2024-09-17T23:28:00.194 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:28:00.194 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:28:00.194 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_25 2024-09-17T23:28:00.194 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=started 2024-09-17T23:28:00.194 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_result 2024-09-17T23:28:00.194 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=stopped 2024-09-17T23:28:00.195 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_25 2024-09-17T23:28:00.195 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ grep -i state 2024-09-17T23:28:00.195 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_instance=cluster1 2024-09-17T23:28:00.195 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:28:00.195 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:28:00.195 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance=cluster1 2024-09-17T23:28:00.195 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local loop_instance 2024-09-17T23:28:00.195 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:28:00.195 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sleep 0 2024-09-17T23:28:00.196 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-17T23:28:00.196 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ seq 0 3 2024-09-17T23:28:00.198 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:28:00.198 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_25 2024-09-17T23:28:00.198 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:0 2024-09-17T23:28:00.198 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:28:00.198 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:28:00.198 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-17T23:28:00.198 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:0 2024-09-17T23:28:00.198 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:28:00.198 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:28:00.198 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:28:00.199 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=0 2024-09-17T23:28:00.199 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-17T23:28:00.199 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:28:00.199 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:28:00.199 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=0 2024-09-17T23:28:00.199 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:28:00.199 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-17T23:28:00.199 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:0 2024-09-17T23:28:00.199 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:28:00.199 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:28:00.199 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-17T23:28:00.199 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:0 2024-09-17T23:28:00.200 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:28:00.200 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:28:00.201 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:28:00.201 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:28:00.201 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-17T23:28:00.201 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:28:00.201 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:28:00.201 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=0 2024-09-17T23:28:00.201 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=0 2024-09-17T23:28:00.201 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-17T23:28:00.221 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:28:00.221 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:28:00.221 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:28:00.221 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_25 2024-09-17T23:28:00.347 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ return 0 2024-09-17T23:28:00.347 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ status_result=' "state": "Replaying"' 2024-09-17T23:28:00.348 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ echo ' "state": "Replaying"' 2024-09-17T23:28:00.348 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep -i Replaying 2024-09-17T23:28:00.348 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: "state": "Replaying" 2024-09-17T23:28:00.350 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_state=started 2024-09-17T23:28:00.350 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test started = started 2024-09-17T23:28:00.350 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:28:00.350 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_25 2024-09-17T23:28:00.350 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:28:00.350 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:28:00.350 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:28:00.350 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_25 2024-09-17T23:28:00.351 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = journal ']' 2024-09-17T23:28:00.351 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = snapshot ']' 2024-09-17T23:28:00.351 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_25 2024-09-17T23:28:00.351 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:28:00.351 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:28:00.351 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:28:00.351 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_25 2024-09-17T23:28:00.351 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster2-mirror-image_25.status 2024-09-17T23:28:00.351 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster2-mirror-image_25.status 2024-09-17T23:28:00.352 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:28:00.353 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_25.status 2024-09-17T23:28:00.353 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster1-mirror-image_25.status 2024-09-17T23:28:00.353 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster1-mirror-image_25.status 2024-09-17T23:28:00.353 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:28:00.353 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_status_log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_25.status 2024-09-17T23:28:00.353 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ mirror_image_snapshot cluster2 mirror image_25 2024-09-17T23:28:00.353 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:28:00.353 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:28:00.353 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_25 2024-09-17T23:28:00.353 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_25 2024-09-17T23:28:01.197 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Snapshot ID: 134 2024-09-17T23:28:01.210 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_25 /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_25.status 2024-09-17T23:28:01.210 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:28:01.210 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:28:01.210 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_25 2024-09-17T23:28:01.210 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_25.status 2024-09-17T23:28:01.210 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_25 --format xml 2024-09-17T23:28:01.210 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:28:01.259 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-17T23:28:01.261 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snapshot_id=134 2024-09-17T23:28:01.261 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:28:01.262 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:28:01.262 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.2 2024-09-17T23:28:01.463 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_25.status 2024-09-17T23:28:01.463 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:28:01.463 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:28:01.463 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_25 2024-09-17T23:28:01.463 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_25.status 2024-09-17T23:28:01.463 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2024-09-17T23:28:01.464 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:28:01.515 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:28:01.515 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:28:01.516 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:28:01.516 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:28:01.516 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:28:01.517 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:28:01.517 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 134 = '' 2024-09-17T23:28:01.517 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:28:01.517 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.4 2024-09-17T23:28:01.918 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_25.status 2024-09-17T23:28:01.918 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:28:01.918 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:28:01.918 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_25 2024-09-17T23:28:01.919 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_25.status 2024-09-17T23:28:01.919 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2024-09-17T23:28:01.919 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:28:01.971 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:28:01.972 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:28:01.973 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:28:01.973 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:28:01.973 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:28:01.974 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:28:01.974 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 134 = '' 2024-09-17T23:28:01.974 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:28:01.974 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.8 2024-09-17T23:28:02.296 INFO:tasks.ceph.cluster2.osd.2.smithi038.stderr:2024-09-17T23:28:02.293+0000 7f1c4b04a640 -1 reset not still connected to 0x561cc9894dd0 2024-09-17T23:28:02.298 INFO:tasks.ceph.cluster2.osd.2.smithi038.stderr:2024-09-17T23:28:02.296+0000 7f1c4b04a640 -1 reset not still connected to 0x561cce7af380 2024-09-17T23:28:02.311 INFO:tasks.ceph.cluster2.osd.2.smithi038.stderr:2024-09-17T23:28:02.308+0000 7f1c4b04a640 -1 reset not still connected to 0x561ccf7fe680 2024-09-17T23:28:02.311 INFO:tasks.ceph.cluster2.osd.2.smithi038.stderr:2024-09-17T23:28:02.308+0000 7f1c4b04a640 -1 reset not still connected to 0x561cd66725b0 2024-09-17T23:28:02.311 INFO:tasks.ceph.cluster2.osd.2.smithi038.stderr:2024-09-17T23:28:02.308+0000 7f1c4b04a640 -1 reset not still connected to 0x561cd80c6340 2024-09-17T23:28:02.316 INFO:tasks.ceph.cluster2.osd.2.smithi038.stderr:2024-09-17T23:28:02.313+0000 7f1c4b04a640 -1 reset not still connected to 0x561cd854ef70 2024-09-17T23:28:02.316 INFO:tasks.ceph.cluster2.osd.2.smithi038.stderr:2024-09-17T23:28:02.313+0000 7f1c4b04a640 -1 reset not still connected to 0x561cd89ba000 2024-09-17T23:28:02.316 INFO:tasks.ceph.cluster2.osd.2.smithi038.stderr:2024-09-17T23:28:02.313+0000 7f1c4b04a640 -1 reset not still connected to 0x561cd9975860 2024-09-17T23:28:02.316 INFO:tasks.ceph.cluster2.osd.2.smithi038.stderr:2024-09-17T23:28:02.313+0000 7f1c4b04a640 -1 reset not still connected to 0x561cda178340 2024-09-17T23:28:02.316 INFO:tasks.ceph.cluster2.osd.2.smithi038.stderr:2024-09-17T23:28:02.313+0000 7f1c4b04a640 -1 reset not still connected to 0x561ce5a8af70 2024-09-17T23:28:02.316 INFO:tasks.ceph.cluster2.osd.2.smithi038.stderr:2024-09-17T23:28:02.313+0000 7f1c4b04a640 -1 reset not still connected to 0x561ce9127a00 2024-09-17T23:28:02.319 INFO:tasks.ceph.cluster2.osd.2.smithi038.stderr:2024-09-17T23:28:02.316+0000 7f1c4b04a640 -1 reset not still connected to 0x561ce952b1e0 2024-09-17T23:28:02.319 INFO:tasks.ceph.cluster2.osd.2.smithi038.stderr:2024-09-17T23:28:02.317+0000 7f1c4b04a640 -1 reset not still connected to 0x561ce9ced110 2024-09-17T23:28:02.319 INFO:tasks.ceph.cluster2.osd.2.smithi038.stderr:2024-09-17T23:28:02.317+0000 7f1c4b04a640 -1 reset not still connected to 0x561cee725c70 2024-09-17T23:28:02.320 INFO:tasks.ceph.cluster2.osd.2.smithi038.stderr:2024-09-17T23:28:02.317+0000 7f1c4b04a640 -1 reset not still connected to 0x561cf092f860 2024-09-17T23:28:02.775 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_25.status 2024-09-17T23:28:02.775 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:28:02.776 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:28:02.776 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_25 2024-09-17T23:28:02.776 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_25.status 2024-09-17T23:28:02.776 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2024-09-17T23:28:02.776 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:28:02.823 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:28:02.825 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id=134 2024-09-17T23:28:02.825 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 134 = 134 2024-09-17T23:28:02.826 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:28:02.826 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snap_present cluster1 mirror image_25 snap 2024-09-17T23:28:02.826 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:28:02.826 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:28:02.826 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_25 2024-09-17T23:28:02.826 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:28:02.826 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:28:02.826 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-17T23:28:02.826 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:28:03.826 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 info mirror/image_25@snap 2024-09-17T23:28:03.874 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'image_25': 2024-09-17T23:28:03.875 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 128 MiB in 32 objects 2024-09-17T23:28:03.875 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:28:03.875 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 2 2024-09-17T23:28:03.875 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 10c6d1793830 2024-09-17T23:28:03.875 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.10c6d1793830 2024-09-17T23:28:03.875 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:28:03.875 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-17T23:28:03.875 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:28:03.875 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:28:03.875 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:20:34 2024 2024-09-17T23:28:03.875 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:20:34 2024 2024-09-17T23:28:03.876 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:20:34 2024 2024-09-17T23:28:03.876 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: protected: False 2024-09-17T23:28:03.876 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:28:03.876 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:28:03.876 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: d504d61f-d7b1-44e8-a7fe-4d46bee040b2 2024-09-17T23:28:03.876 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: false 2024-09-17T23:28:03.879 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:28:03.879 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ compare_image_snaps mirror image_25 snap 2024-09-17T23:28:03.879 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:28:03.879 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_25 2024-09-17T23:28:03.879 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:28:03.879 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local ret=0 2024-09-17T23:28:03.879 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local rmt_export=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_25.export 2024-09-17T23:28:03.879 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local loc_export=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_25.export 2024-09-17T23:28:03.879 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_25.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_25.export 2024-09-17T23:28:03.879 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror export image_25@snap /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_25.export 2024-09-17T23:28:04.047 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-17T23:28:04.054 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 -p mirror export image_25@snap /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_25.export 2024-09-17T23:28:04.304 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-17T23:28:04.310 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ cmp /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_25.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_25.export 2024-09-17T23:28:04.373 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_25.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_25.export 2024-09-17T23:28:04.404 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:28:04.404 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:28:04.404 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_26 2024-09-17T23:28:04.404 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_snap cluster2 mirror image_26 snap 2024-09-17T23:28:04.404 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:28:04.404 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:28:04.404 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_26 2024-09-17T23:28:04.405 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:28:04.405 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror snap create image_26@snap --debug-rbd=20 --debug-journaler=20 2024-09-17T23:28:06.223 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_started cluster1 mirror image_26 2024-09-17T23:28:06.223 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:28:06.223 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:28:06.223 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_26 2024-09-17T23:28:06.223 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_state cluster1 mirror image_26 started 2024-09-17T23:28:06.223 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:28:06.223 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:28:06.223 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_26 2024-09-17T23:28:06.223 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=started 2024-09-17T23:28:06.223 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:28:06.223 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:28:06.223 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:28:07.224 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_replay_state cluster1 mirror image_26 started 2024-09-17T23:28:07.224 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:28:07.224 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:28:07.224 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_26 2024-09-17T23:28:07.224 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=started 2024-09-17T23:28:07.224 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_result 2024-09-17T23:28:07.225 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=stopped 2024-09-17T23:28:07.225 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_26 2024-09-17T23:28:07.225 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_instance=cluster1 2024-09-17T23:28:07.225 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:28:07.226 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:28:07.226 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ grep -i state 2024-09-17T23:28:07.226 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance=cluster1 2024-09-17T23:28:07.226 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local loop_instance 2024-09-17T23:28:07.226 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:28:07.226 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sleep 0 2024-09-17T23:28:07.226 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-17T23:28:07.227 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ seq 0 3 2024-09-17T23:28:07.229 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:28:07.229 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_26 2024-09-17T23:28:07.229 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:0 2024-09-17T23:28:07.229 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:28:07.229 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:28:07.229 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-17T23:28:07.229 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:0 2024-09-17T23:28:07.229 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:28:07.229 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:28:07.230 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:28:07.230 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=0 2024-09-17T23:28:07.230 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-17T23:28:07.230 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:28:07.230 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:28:07.230 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=0 2024-09-17T23:28:07.230 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:28:07.230 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-17T23:28:07.230 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:0 2024-09-17T23:28:07.231 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:28:07.231 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:28:07.231 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-17T23:28:07.231 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:0 2024-09-17T23:28:07.231 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:28:07.231 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:28:07.231 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:28:07.231 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:28:07.231 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-17T23:28:07.232 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:28:07.232 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:28:07.232 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=0 2024-09-17T23:28:07.232 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=0 2024-09-17T23:28:07.232 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-17T23:28:07.251 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:28:07.252 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:28:07.252 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:28:07.252 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_26 2024-09-17T23:28:07.378 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ return 0 2024-09-17T23:28:07.379 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ status_result=' "state": "Replaying"' 2024-09-17T23:28:07.379 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ echo ' "state": "Replaying"' 2024-09-17T23:28:07.379 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep -i Replaying 2024-09-17T23:28:07.380 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: "state": "Replaying" 2024-09-17T23:28:07.381 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_state=started 2024-09-17T23:28:07.381 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test started = started 2024-09-17T23:28:07.382 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:28:07.382 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_26 2024-09-17T23:28:07.382 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:28:07.382 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:28:07.382 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:28:07.382 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_26 2024-09-17T23:28:07.382 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = journal ']' 2024-09-17T23:28:07.382 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = snapshot ']' 2024-09-17T23:28:07.382 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_26 2024-09-17T23:28:07.382 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:28:07.382 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:28:07.382 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:28:07.382 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_26 2024-09-17T23:28:07.383 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster2-mirror-image_26.status 2024-09-17T23:28:07.383 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster2-mirror-image_26.status 2024-09-17T23:28:07.383 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:28:07.383 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_26.status 2024-09-17T23:28:07.383 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster1-mirror-image_26.status 2024-09-17T23:28:07.383 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster1-mirror-image_26.status 2024-09-17T23:28:07.384 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:28:07.384 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_status_log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_26.status 2024-09-17T23:28:07.384 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ mirror_image_snapshot cluster2 mirror image_26 2024-09-17T23:28:07.385 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:28:07.385 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:28:07.385 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_26 2024-09-17T23:28:07.385 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_26 2024-09-17T23:28:08.234 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Snapshot ID: 136 2024-09-17T23:28:08.245 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_26 /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_26.status 2024-09-17T23:28:08.246 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:28:08.246 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:28:08.246 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_26 2024-09-17T23:28:08.246 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_26.status 2024-09-17T23:28:08.246 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_26 --format xml 2024-09-17T23:28:08.246 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:28:08.293 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-17T23:28:08.294 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snapshot_id=136 2024-09-17T23:28:08.295 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:28:08.295 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:28:08.295 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.2 2024-09-17T23:28:08.496 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_26.status 2024-09-17T23:28:08.496 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:28:08.496 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:28:08.496 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_26 2024-09-17T23:28:08.496 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_26.status 2024-09-17T23:28:08.496 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2024-09-17T23:28:08.496 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:28:08.546 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:28:08.547 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:28:08.549 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:28:08.549 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:28:08.549 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:28:08.549 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:28:08.549 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 136 = '' 2024-09-17T23:28:08.549 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:28:08.549 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.4 2024-09-17T23:28:08.950 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_26.status 2024-09-17T23:28:08.950 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:28:08.951 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:28:08.951 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_26 2024-09-17T23:28:08.951 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_26.status 2024-09-17T23:28:08.951 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2024-09-17T23:28:08.951 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:28:09.001 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:28:09.001 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:28:09.002 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:28:09.002 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:28:09.002 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:28:09.003 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:28:09.003 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 136 = '' 2024-09-17T23:28:09.003 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:28:09.003 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.8 2024-09-17T23:28:09.804 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_26.status 2024-09-17T23:28:09.804 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:28:09.804 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:28:09.804 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_26 2024-09-17T23:28:09.805 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_26.status 2024-09-17T23:28:09.805 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2024-09-17T23:28:09.805 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:28:09.853 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:28:09.855 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id=136 2024-09-17T23:28:09.855 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 136 = 136 2024-09-17T23:28:09.855 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:28:09.855 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snap_present cluster1 mirror image_26 snap 2024-09-17T23:28:09.855 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:28:09.855 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:28:09.855 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_26 2024-09-17T23:28:09.855 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:28:09.856 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:28:09.856 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-17T23:28:09.856 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:28:10.856 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 info mirror/image_26@snap 2024-09-17T23:28:10.906 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'image_26': 2024-09-17T23:28:10.906 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 128 MiB in 32 objects 2024-09-17T23:28:10.906 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:28:10.906 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 2 2024-09-17T23:28:10.907 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 10c69633bb01 2024-09-17T23:28:10.907 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.10c69633bb01 2024-09-17T23:28:10.907 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:28:10.907 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-17T23:28:10.907 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:28:10.907 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:28:10.907 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:20:42 2024 2024-09-17T23:28:10.907 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:20:42 2024 2024-09-17T23:28:10.907 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:20:42 2024 2024-09-17T23:28:10.907 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: protected: False 2024-09-17T23:28:10.907 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:28:10.907 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:28:10.907 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: abfb6ed3-8b33-429f-bf1d-a7fe92644887 2024-09-17T23:28:10.907 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: false 2024-09-17T23:28:10.910 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:28:10.910 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ compare_image_snaps mirror image_26 snap 2024-09-17T23:28:10.910 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:28:10.910 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_26 2024-09-17T23:28:10.910 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:28:10.910 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local ret=0 2024-09-17T23:28:10.911 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local rmt_export=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_26.export 2024-09-17T23:28:10.911 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local loc_export=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_26.export 2024-09-17T23:28:10.911 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_26.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_26.export 2024-09-17T23:28:10.911 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror export image_26@snap /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_26.export 2024-09-17T23:28:11.089 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-17T23:28:11.095 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 -p mirror export image_26@snap /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_26.export 2024-09-17T23:28:11.342 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-17T23:28:11.350 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ cmp /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_26.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_26.export 2024-09-17T23:28:11.413 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_26.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_26.export 2024-09-17T23:28:11.444 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:28:11.445 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:28:11.445 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_27 2024-09-17T23:28:11.445 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_snap cluster2 mirror image_27 snap 2024-09-17T23:28:11.445 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:28:11.445 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:28:11.445 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_27 2024-09-17T23:28:11.445 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:28:11.445 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror snap create image_27@snap --debug-rbd=20 --debug-journaler=20 2024-09-17T23:28:13.264 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_started cluster1 mirror image_27 2024-09-17T23:28:13.265 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:28:13.265 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:28:13.265 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_27 2024-09-17T23:28:13.265 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_state cluster1 mirror image_27 started 2024-09-17T23:28:13.265 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:28:13.265 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:28:13.265 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_27 2024-09-17T23:28:13.265 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=started 2024-09-17T23:28:13.265 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:28:13.265 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:28:13.265 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:28:14.266 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_replay_state cluster1 mirror image_27 started 2024-09-17T23:28:14.266 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:28:14.266 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:28:14.266 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_27 2024-09-17T23:28:14.266 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=started 2024-09-17T23:28:14.266 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_result 2024-09-17T23:28:14.266 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=stopped 2024-09-17T23:28:14.267 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_27 2024-09-17T23:28:14.268 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_instance=cluster1 2024-09-17T23:28:14.268 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:28:14.268 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ grep -i state 2024-09-17T23:28:14.268 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:28:14.268 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance=cluster1 2024-09-17T23:28:14.268 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local loop_instance 2024-09-17T23:28:14.268 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:28:14.268 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sleep 0 2024-09-17T23:28:14.268 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-17T23:28:14.268 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ seq 0 3 2024-09-17T23:28:14.270 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:28:14.270 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_27 2024-09-17T23:28:14.270 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:0 2024-09-17T23:28:14.270 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:28:14.270 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:28:14.270 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-17T23:28:14.270 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:0 2024-09-17T23:28:14.271 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:28:14.271 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:28:14.271 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:28:14.271 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=0 2024-09-17T23:28:14.271 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-17T23:28:14.271 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:28:14.271 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:28:14.271 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=0 2024-09-17T23:28:14.271 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:28:14.271 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-17T23:28:14.271 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:0 2024-09-17T23:28:14.271 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:28:14.272 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:28:14.272 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-17T23:28:14.272 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:0 2024-09-17T23:28:14.273 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:28:14.273 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:28:14.273 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:28:14.274 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:28:14.274 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-17T23:28:14.274 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:28:14.274 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:28:14.274 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=0 2024-09-17T23:28:14.274 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=0 2024-09-17T23:28:14.274 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-17T23:28:14.302 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:28:14.302 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:28:14.302 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:28:14.302 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_27 2024-09-17T23:28:14.439 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ return 0 2024-09-17T23:28:14.439 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ status_result=' "state": "Replaying"' 2024-09-17T23:28:14.439 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ echo ' "state": "Replaying"' 2024-09-17T23:28:14.440 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep -i Replaying 2024-09-17T23:28:14.440 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: "state": "Replaying" 2024-09-17T23:28:14.442 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_state=started 2024-09-17T23:28:14.442 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test started = started 2024-09-17T23:28:14.442 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:28:14.442 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_27 2024-09-17T23:28:14.442 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:28:14.442 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:28:14.442 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:28:14.442 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_27 2024-09-17T23:28:14.442 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = journal ']' 2024-09-17T23:28:14.443 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = snapshot ']' 2024-09-17T23:28:14.443 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_27 2024-09-17T23:28:14.443 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:28:14.443 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:28:14.443 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:28:14.443 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_27 2024-09-17T23:28:14.443 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster2-mirror-image_27.status 2024-09-17T23:28:14.443 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster2-mirror-image_27.status 2024-09-17T23:28:14.443 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:28:14.444 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_27.status 2024-09-17T23:28:14.444 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster1-mirror-image_27.status 2024-09-17T23:28:14.444 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster1-mirror-image_27.status 2024-09-17T23:28:14.444 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:28:14.445 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_status_log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_27.status 2024-09-17T23:28:14.445 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ mirror_image_snapshot cluster2 mirror image_27 2024-09-17T23:28:14.445 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:28:14.445 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:28:14.445 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_27 2024-09-17T23:28:14.445 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_27 2024-09-17T23:28:15.271 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Snapshot ID: 138 2024-09-17T23:28:15.285 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_27 /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_27.status 2024-09-17T23:28:15.285 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:28:15.285 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:28:15.285 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_27 2024-09-17T23:28:15.285 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_27.status 2024-09-17T23:28:15.285 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_27 --format xml 2024-09-17T23:28:15.285 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:28:15.333 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-17T23:28:15.335 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snapshot_id=138 2024-09-17T23:28:15.335 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:28:15.335 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:28:15.335 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.2 2024-09-17T23:28:15.537 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_27.status 2024-09-17T23:28:15.537 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:28:15.537 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:28:15.537 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_27 2024-09-17T23:28:15.537 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_27.status 2024-09-17T23:28:15.537 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2024-09-17T23:28:15.538 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:28:15.591 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:28:15.593 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id=61 2024-09-17T23:28:15.593 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 138 = 61 2024-09-17T23:28:15.593 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:28:15.593 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.4 2024-09-17T23:28:15.994 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_27.status 2024-09-17T23:28:15.995 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:28:15.995 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:28:15.995 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_27 2024-09-17T23:28:15.995 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_27.status 2024-09-17T23:28:15.995 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2024-09-17T23:28:15.995 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:28:16.047 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:28:16.049 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id=61 2024-09-17T23:28:16.049 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 138 = 61 2024-09-17T23:28:16.049 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:28:16.049 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.8 2024-09-17T23:28:16.851 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_27.status 2024-09-17T23:28:16.851 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:28:16.851 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:28:16.851 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_27 2024-09-17T23:28:16.851 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_27.status 2024-09-17T23:28:16.851 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2024-09-17T23:28:16.851 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:28:16.904 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:28:16.904 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:28:16.906 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:28:16.906 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:28:16.906 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:28:16.906 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:28:16.906 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 138 = '' 2024-09-17T23:28:16.906 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:28:16.907 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1.6 2024-09-17T23:28:18.508 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_27.status 2024-09-17T23:28:18.508 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:28:18.508 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:28:18.508 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_27 2024-09-17T23:28:18.508 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_27.status 2024-09-17T23:28:18.508 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2024-09-17T23:28:18.508 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:28:18.564 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:28:18.567 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id=138 2024-09-17T23:28:18.567 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 138 = 138 2024-09-17T23:28:18.567 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:28:18.567 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snap_present cluster1 mirror image_27 snap 2024-09-17T23:28:18.567 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:28:18.567 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:28:18.567 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_27 2024-09-17T23:28:18.567 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:28:18.567 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:28:18.567 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-17T23:28:18.567 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:28:19.567 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 info mirror/image_27@snap 2024-09-17T23:28:19.617 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'image_27': 2024-09-17T23:28:19.617 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 128 MiB in 32 objects 2024-09-17T23:28:19.617 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:28:19.617 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 2 2024-09-17T23:28:19.617 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 10c698066f8f 2024-09-17T23:28:19.618 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.10c698066f8f 2024-09-17T23:28:19.618 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:28:19.618 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-17T23:28:19.618 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:28:19.618 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:28:19.618 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:20:51 2024 2024-09-17T23:28:19.618 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:20:51 2024 2024-09-17T23:28:19.618 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:20:51 2024 2024-09-17T23:28:19.618 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: protected: False 2024-09-17T23:28:19.618 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:28:19.618 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:28:19.618 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: 3340f84d-5843-4769-b7b7-cd6b8eee8192 2024-09-17T23:28:19.618 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: false 2024-09-17T23:28:19.621 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:28:19.621 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ compare_image_snaps mirror image_27 snap 2024-09-17T23:28:19.621 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:28:19.621 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_27 2024-09-17T23:28:19.621 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:28:19.621 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local ret=0 2024-09-17T23:28:19.621 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local rmt_export=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_27.export 2024-09-17T23:28:19.621 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local loc_export=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_27.export 2024-09-17T23:28:19.622 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_27.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_27.export 2024-09-17T23:28:19.622 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror export image_27@snap /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_27.export 2024-09-17T23:28:19.796 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-17T23:28:19.803 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 -p mirror export image_27@snap /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_27.export 2024-09-17T23:28:20.030 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-17T23:28:20.037 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ cmp /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_27.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_27.export 2024-09-17T23:28:20.100 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_27.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_27.export 2024-09-17T23:28:20.133 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:28:20.133 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:28:20.133 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_28 2024-09-17T23:28:20.133 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_snap cluster2 mirror image_28 snap 2024-09-17T23:28:20.133 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:28:20.134 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:28:20.134 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_28 2024-09-17T23:28:20.134 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:28:20.134 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror snap create image_28@snap --debug-rbd=20 --debug-journaler=20 2024-09-17T23:28:21.989 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_started cluster1 mirror image_28 2024-09-17T23:28:21.989 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:28:21.989 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:28:21.989 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_28 2024-09-17T23:28:21.989 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_state cluster1 mirror image_28 started 2024-09-17T23:28:21.990 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:28:21.990 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:28:21.990 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_28 2024-09-17T23:28:21.990 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=started 2024-09-17T23:28:21.990 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:28:21.990 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:28:21.990 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:28:22.990 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_replay_state cluster1 mirror image_28 started 2024-09-17T23:28:22.991 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:28:22.991 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:28:22.991 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_28 2024-09-17T23:28:22.991 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=started 2024-09-17T23:28:22.991 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_result 2024-09-17T23:28:22.991 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=stopped 2024-09-17T23:28:22.992 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_28 2024-09-17T23:28:22.992 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_instance=cluster1 2024-09-17T23:28:22.992 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:28:22.992 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ grep -i state 2024-09-17T23:28:22.992 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:28:22.992 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance=cluster1 2024-09-17T23:28:22.992 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local loop_instance 2024-09-17T23:28:22.992 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:28:22.993 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sleep 0 2024-09-17T23:28:22.993 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-17T23:28:22.995 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ seq 0 3 2024-09-17T23:28:22.995 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:28:22.995 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_28 2024-09-17T23:28:22.995 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:0 2024-09-17T23:28:22.995 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:28:22.995 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:28:22.995 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-17T23:28:22.995 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:0 2024-09-17T23:28:22.995 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:28:22.995 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:28:22.996 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:28:22.996 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=0 2024-09-17T23:28:22.996 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-17T23:28:22.996 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:28:22.996 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:28:22.996 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=0 2024-09-17T23:28:22.996 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:28:22.996 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-17T23:28:22.996 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:0 2024-09-17T23:28:22.996 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:28:22.996 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:28:22.996 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-17T23:28:22.996 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:0 2024-09-17T23:28:22.996 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:28:22.996 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:28:22.997 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:28:22.997 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:28:22.997 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-17T23:28:22.997 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:28:22.997 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:28:22.997 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=0 2024-09-17T23:28:22.998 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=0 2024-09-17T23:28:22.998 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-17T23:28:23.017 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:28:23.017 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:28:23.017 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:28:23.017 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_28 2024-09-17T23:28:23.144 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ return 0 2024-09-17T23:28:23.144 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ status_result=' "state": "Replaying"' 2024-09-17T23:28:23.145 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ echo ' "state": "Replaying"' 2024-09-17T23:28:23.145 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep -i Replaying 2024-09-17T23:28:23.145 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: "state": "Replaying" 2024-09-17T23:28:23.147 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_state=started 2024-09-17T23:28:23.147 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test started = started 2024-09-17T23:28:23.147 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:28:23.147 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_28 2024-09-17T23:28:23.147 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:28:23.147 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:28:23.147 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:28:23.147 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_28 2024-09-17T23:28:23.147 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = journal ']' 2024-09-17T23:28:23.147 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = snapshot ']' 2024-09-17T23:28:23.147 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_28 2024-09-17T23:28:23.147 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:28:23.148 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:28:23.148 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:28:23.148 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_28 2024-09-17T23:28:23.148 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster2-mirror-image_28.status 2024-09-17T23:28:23.148 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster2-mirror-image_28.status 2024-09-17T23:28:23.148 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:28:23.149 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_28.status 2024-09-17T23:28:23.149 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster1-mirror-image_28.status 2024-09-17T23:28:23.149 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster1-mirror-image_28.status 2024-09-17T23:28:23.149 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:28:23.150 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_status_log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_28.status 2024-09-17T23:28:23.150 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ mirror_image_snapshot cluster2 mirror image_28 2024-09-17T23:28:23.150 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:28:23.150 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:28:23.150 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_28 2024-09-17T23:28:23.150 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_28 2024-09-17T23:28:24.000 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Snapshot ID: 140 2024-09-17T23:28:24.012 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_28 /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_28.status 2024-09-17T23:28:24.013 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:28:24.013 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:28:24.013 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_28 2024-09-17T23:28:24.013 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_28.status 2024-09-17T23:28:24.013 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_28 --format xml 2024-09-17T23:28:24.013 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:28:24.061 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-17T23:28:24.063 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snapshot_id=140 2024-09-17T23:28:24.063 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:28:24.063 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:28:24.063 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.2 2024-09-17T23:28:24.264 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_28.status 2024-09-17T23:28:24.264 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:28:24.264 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:28:24.265 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_28 2024-09-17T23:28:24.265 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_28.status 2024-09-17T23:28:24.265 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2024-09-17T23:28:24.265 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:28:24.314 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:28:24.315 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:28:24.316 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:28:24.316 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:28:24.316 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:28:24.317 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:28:24.317 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 140 = '' 2024-09-17T23:28:24.317 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:28:24.317 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.4 2024-09-17T23:28:24.718 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_28.status 2024-09-17T23:28:24.718 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:28:24.718 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:28:24.719 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_28 2024-09-17T23:28:24.719 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_28.status 2024-09-17T23:28:24.719 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2024-09-17T23:28:24.719 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:28:24.773 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:28:24.773 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:28:24.775 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:28:24.775 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:28:24.775 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:28:24.775 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:28:24.775 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 140 = '' 2024-09-17T23:28:24.776 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:28:24.776 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.8 2024-09-17T23:28:25.577 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_28.status 2024-09-17T23:28:25.577 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:28:25.577 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:28:25.577 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_28 2024-09-17T23:28:25.577 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_28.status 2024-09-17T23:28:25.577 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2024-09-17T23:28:25.577 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:28:25.825 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:28:25.828 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id=140 2024-09-17T23:28:25.828 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 140 = 140 2024-09-17T23:28:25.828 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:28:25.828 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snap_present cluster1 mirror image_28 snap 2024-09-17T23:28:25.828 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:28:25.828 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:28:25.828 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_28 2024-09-17T23:28:25.828 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:28:25.828 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:28:25.828 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-17T23:28:25.828 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:28:26.829 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 info mirror/image_28@snap 2024-09-17T23:28:26.876 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'image_28': 2024-09-17T23:28:26.876 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 128 MiB in 32 objects 2024-09-17T23:28:26.876 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:28:26.877 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 2 2024-09-17T23:28:26.877 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 10c6e9d44423 2024-09-17T23:28:26.877 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.10c6e9d44423 2024-09-17T23:28:26.877 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:28:26.877 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-17T23:28:26.877 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:28:26.877 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:28:26.877 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:21:02 2024 2024-09-17T23:28:26.877 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:21:02 2024 2024-09-17T23:28:26.877 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:21:02 2024 2024-09-17T23:28:26.877 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: protected: False 2024-09-17T23:28:26.877 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:28:26.877 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:28:26.878 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: aa5b5df7-2c98-4b35-a04c-18d3b14fb4a4 2024-09-17T23:28:26.878 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: false 2024-09-17T23:28:26.880 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:28:26.880 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ compare_image_snaps mirror image_28 snap 2024-09-17T23:28:26.880 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:28:26.881 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_28 2024-09-17T23:28:26.881 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:28:26.881 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local ret=0 2024-09-17T23:28:26.881 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local rmt_export=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_28.export 2024-09-17T23:28:26.881 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local loc_export=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_28.export 2024-09-17T23:28:26.881 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_28.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_28.export 2024-09-17T23:28:26.881 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror export image_28@snap /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_28.export 2024-09-17T23:28:27.072 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-17T23:28:27.081 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 -p mirror export image_28@snap /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_28.export 2024-09-17T23:28:27.297 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-17T23:28:27.305 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ cmp /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_28.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_28.export 2024-09-17T23:28:27.368 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_28.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_28.export 2024-09-17T23:28:27.399 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:28:27.400 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:28:27.400 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_29 2024-09-17T23:28:27.400 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_snap cluster2 mirror image_29 snap 2024-09-17T23:28:27.400 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:28:27.400 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:28:27.400 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_29 2024-09-17T23:28:27.400 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:28:27.400 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror snap create image_29@snap --debug-rbd=20 --debug-journaler=20 2024-09-17T23:28:29.036 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_started cluster1 mirror image_29 2024-09-17T23:28:29.036 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:28:29.036 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:28:29.036 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_29 2024-09-17T23:28:29.037 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_state cluster1 mirror image_29 started 2024-09-17T23:28:29.037 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:28:29.037 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:28:29.037 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_29 2024-09-17T23:28:29.037 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=started 2024-09-17T23:28:29.037 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:28:29.037 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:28:29.037 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:28:30.037 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_replay_state cluster1 mirror image_29 started 2024-09-17T23:28:30.038 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:28:30.038 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:28:30.038 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_29 2024-09-17T23:28:30.038 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=started 2024-09-17T23:28:30.038 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_result 2024-09-17T23:28:30.038 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=stopped 2024-09-17T23:28:30.039 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_29 2024-09-17T23:28:30.039 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_instance=cluster1 2024-09-17T23:28:30.039 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ grep -i state 2024-09-17T23:28:30.039 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:28:30.039 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:28:30.039 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance=cluster1 2024-09-17T23:28:30.040 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local loop_instance 2024-09-17T23:28:30.040 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:28:30.040 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sleep 0 2024-09-17T23:28:30.040 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-17T23:28:30.041 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ seq 0 3 2024-09-17T23:28:30.041 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:28:30.041 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_29 2024-09-17T23:28:30.042 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:0 2024-09-17T23:28:30.042 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:28:30.042 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:28:30.042 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-17T23:28:30.042 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:0 2024-09-17T23:28:30.042 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:28:30.042 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:28:30.042 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:28:30.042 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=0 2024-09-17T23:28:30.042 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-17T23:28:30.042 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:28:30.042 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:28:30.042 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=0 2024-09-17T23:28:30.042 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:28:30.044 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-17T23:28:30.044 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:0 2024-09-17T23:28:30.045 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:28:30.045 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:28:30.045 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-17T23:28:30.045 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:0 2024-09-17T23:28:30.045 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:28:30.045 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:28:30.045 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:28:30.045 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:28:30.045 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-17T23:28:30.045 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:28:30.045 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:28:30.045 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=0 2024-09-17T23:28:30.045 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=0 2024-09-17T23:28:30.046 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-17T23:28:30.065 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:28:30.065 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:28:30.065 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:28:30.065 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_29 2024-09-17T23:28:30.193 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ return 0 2024-09-17T23:28:30.193 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ status_result=' "state": "Replaying"' 2024-09-17T23:28:30.194 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ echo ' "state": "Replaying"' 2024-09-17T23:28:30.194 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep -i Replaying 2024-09-17T23:28:30.194 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: "state": "Replaying" 2024-09-17T23:28:30.196 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_state=started 2024-09-17T23:28:30.196 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test started = started 2024-09-17T23:28:30.196 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:28:30.196 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_29 2024-09-17T23:28:30.196 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:28:30.196 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:28:30.196 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:28:30.196 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_29 2024-09-17T23:28:30.196 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = journal ']' 2024-09-17T23:28:30.197 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = snapshot ']' 2024-09-17T23:28:30.197 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_29 2024-09-17T23:28:30.197 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:28:30.197 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:28:30.197 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:28:30.197 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_29 2024-09-17T23:28:30.197 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster2-mirror-image_29.status 2024-09-17T23:28:30.197 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster2-mirror-image_29.status 2024-09-17T23:28:30.197 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:28:30.198 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_29.status 2024-09-17T23:28:30.198 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster1-mirror-image_29.status 2024-09-17T23:28:30.198 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster1-mirror-image_29.status 2024-09-17T23:28:30.198 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:28:30.199 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_status_log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_29.status 2024-09-17T23:28:30.199 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ mirror_image_snapshot cluster2 mirror image_29 2024-09-17T23:28:30.199 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:28:30.199 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:28:30.199 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_29 2024-09-17T23:28:30.199 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_29 2024-09-17T23:28:31.045 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Snapshot ID: 142 2024-09-17T23:28:31.058 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_29 /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_29.status 2024-09-17T23:28:31.058 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:28:31.058 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:28:31.058 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_29 2024-09-17T23:28:31.058 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_29.status 2024-09-17T23:28:31.059 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_29 --format xml 2024-09-17T23:28:31.059 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:28:31.108 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-17T23:28:31.110 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snapshot_id=142 2024-09-17T23:28:31.110 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:28:31.110 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:28:31.110 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.2 2024-09-17T23:28:31.311 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_29.status 2024-09-17T23:28:31.311 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:28:31.312 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:28:31.312 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_29 2024-09-17T23:28:31.312 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_29.status 2024-09-17T23:28:31.312 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2024-09-17T23:28:31.312 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:28:31.361 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:28:31.362 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:28:31.363 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:28:31.364 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:28:31.364 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:28:31.364 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:28:31.364 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 142 = '' 2024-09-17T23:28:31.364 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:28:31.365 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.4 2024-09-17T23:28:31.765 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_29.status 2024-09-17T23:28:31.765 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:28:31.765 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:28:31.765 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_29 2024-09-17T23:28:31.765 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_29.status 2024-09-17T23:28:31.765 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2024-09-17T23:28:31.765 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:28:31.818 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:28:31.818 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:28:31.819 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:28:31.820 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:28:31.820 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:28:31.820 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:28:31.820 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 142 = '' 2024-09-17T23:28:31.821 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:28:31.821 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.8 2024-09-17T23:28:32.621 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_29.status 2024-09-17T23:28:32.622 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:28:32.622 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:28:32.622 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_29 2024-09-17T23:28:32.622 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_29.status 2024-09-17T23:28:32.622 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2024-09-17T23:28:32.622 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:28:32.669 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:28:32.671 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id=142 2024-09-17T23:28:32.671 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 142 = 142 2024-09-17T23:28:32.671 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:28:32.672 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snap_present cluster1 mirror image_29 snap 2024-09-17T23:28:32.672 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:28:32.672 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:28:32.672 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_29 2024-09-17T23:28:32.672 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:28:32.672 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:28:32.672 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-17T23:28:32.672 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:28:33.673 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 info mirror/image_29@snap 2024-09-17T23:28:33.721 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'image_29': 2024-09-17T23:28:33.721 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 128 MiB in 32 objects 2024-09-17T23:28:33.721 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:28:33.722 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 2 2024-09-17T23:28:33.722 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 1210dcc24aae 2024-09-17T23:28:33.722 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.1210dcc24aae 2024-09-17T23:28:33.722 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:28:33.722 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-17T23:28:33.722 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:28:33.722 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:28:33.722 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:21:13 2024 2024-09-17T23:28:33.722 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:21:13 2024 2024-09-17T23:28:33.722 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:21:13 2024 2024-09-17T23:28:33.722 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: protected: False 2024-09-17T23:28:33.722 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:28:33.722 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:28:33.723 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: 5ce3fe34-b38d-4dad-9e4f-b5cdc9405a96 2024-09-17T23:28:33.723 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: false 2024-09-17T23:28:33.725 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:28:33.725 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ compare_image_snaps mirror image_29 snap 2024-09-17T23:28:33.725 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:28:33.725 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_29 2024-09-17T23:28:33.725 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:28:33.726 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local ret=0 2024-09-17T23:28:33.726 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local rmt_export=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_29.export 2024-09-17T23:28:33.726 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local loc_export=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_29.export 2024-09-17T23:28:33.726 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_29.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_29.export 2024-09-17T23:28:33.726 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror export image_29@snap /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_29.export 2024-09-17T23:28:33.900 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-17T23:28:33.908 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 -p mirror export image_29@snap /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_29.export 2024-09-17T23:28:34.121 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-17T23:28:34.128 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ cmp /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_29.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_29.export 2024-09-17T23:28:34.190 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_29.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_29.export 2024-09-17T23:28:34.222 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:28:34.222 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:28:34.222 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_30 2024-09-17T23:28:34.222 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_snap cluster2 mirror image_30 snap 2024-09-17T23:28:34.222 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:28:34.222 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:28:34.222 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_30 2024-09-17T23:28:34.222 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:28:34.223 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror snap create image_30@snap --debug-rbd=20 --debug-journaler=20 2024-09-17T23:28:36.058 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_started cluster1 mirror image_30 2024-09-17T23:28:36.058 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:28:36.058 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:28:36.058 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_30 2024-09-17T23:28:36.058 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_state cluster1 mirror image_30 started 2024-09-17T23:28:36.058 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:28:36.058 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:28:36.058 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_30 2024-09-17T23:28:36.058 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=started 2024-09-17T23:28:36.059 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:28:36.059 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:28:36.059 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:28:37.059 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_replay_state cluster1 mirror image_30 started 2024-09-17T23:28:37.059 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:28:37.059 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:28:37.059 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_30 2024-09-17T23:28:37.059 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=started 2024-09-17T23:28:37.059 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_result 2024-09-17T23:28:37.060 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=stopped 2024-09-17T23:28:37.061 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_30 2024-09-17T23:28:37.061 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_instance=cluster1 2024-09-17T23:28:37.061 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:28:37.061 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ grep -i state 2024-09-17T23:28:37.061 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:28:37.061 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance=cluster1 2024-09-17T23:28:37.061 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local loop_instance 2024-09-17T23:28:37.061 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:28:37.061 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sleep 0 2024-09-17T23:28:37.061 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-17T23:28:37.062 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ seq 0 3 2024-09-17T23:28:37.064 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:28:37.064 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_30 2024-09-17T23:28:37.064 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:0 2024-09-17T23:28:37.064 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:28:37.064 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:28:37.064 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-17T23:28:37.064 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:0 2024-09-17T23:28:37.064 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:28:37.064 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:28:37.064 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:28:37.065 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=0 2024-09-17T23:28:37.065 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-17T23:28:37.065 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:28:37.065 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:28:37.065 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=0 2024-09-17T23:28:37.065 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:28:37.065 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-17T23:28:37.065 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:0 2024-09-17T23:28:37.065 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:28:37.065 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:28:37.065 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-17T23:28:37.065 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:0 2024-09-17T23:28:37.065 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:28:37.066 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:28:37.066 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:28:37.066 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:28:37.066 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-17T23:28:37.066 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:28:37.066 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:28:37.066 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=0 2024-09-17T23:28:37.066 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=0 2024-09-17T23:28:37.066 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-17T23:28:37.086 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:28:37.086 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:28:37.086 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:28:37.086 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_30 2024-09-17T23:28:37.214 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ return 0 2024-09-17T23:28:37.215 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ status_result=' "state": "Replaying"' 2024-09-17T23:28:37.215 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ echo ' "state": "Replaying"' 2024-09-17T23:28:37.215 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep -i Replaying 2024-09-17T23:28:37.216 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: "state": "Replaying" 2024-09-17T23:28:37.217 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_state=started 2024-09-17T23:28:37.217 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test started = started 2024-09-17T23:28:37.217 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:28:37.217 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_30 2024-09-17T23:28:37.217 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:28:37.218 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:28:37.218 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:28:37.218 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_30 2024-09-17T23:28:37.218 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = journal ']' 2024-09-17T23:28:37.218 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = snapshot ']' 2024-09-17T23:28:37.218 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_30 2024-09-17T23:28:37.218 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:28:37.218 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:28:37.218 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:28:37.218 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_30 2024-09-17T23:28:37.218 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster2-mirror-image_30.status 2024-09-17T23:28:37.218 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster2-mirror-image_30.status 2024-09-17T23:28:37.218 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:28:37.219 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_30.status 2024-09-17T23:28:37.220 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster1-mirror-image_30.status 2024-09-17T23:28:37.220 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster1-mirror-image_30.status 2024-09-17T23:28:37.220 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:28:37.220 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_status_log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_30.status 2024-09-17T23:28:37.220 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ mirror_image_snapshot cluster2 mirror image_30 2024-09-17T23:28:37.220 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:28:37.220 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:28:37.220 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_30 2024-09-17T23:28:37.220 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_30 2024-09-17T23:28:38.074 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Snapshot ID: 144 2024-09-17T23:28:38.087 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_30 /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_30.status 2024-09-17T23:28:38.087 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:28:38.087 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:28:38.087 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_30 2024-09-17T23:28:38.087 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_30.status 2024-09-17T23:28:38.087 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_30 --format xml 2024-09-17T23:28:38.087 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:28:38.135 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-17T23:28:38.138 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snapshot_id=144 2024-09-17T23:28:38.138 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:28:38.138 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:28:38.138 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.2 2024-09-17T23:28:38.339 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_30.status 2024-09-17T23:28:38.339 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:28:38.339 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:28:38.339 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_30 2024-09-17T23:28:38.339 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_30.status 2024-09-17T23:28:38.339 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2024-09-17T23:28:38.340 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:28:38.389 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:28:38.389 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:28:38.390 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:28:38.390 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:28:38.390 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:28:38.391 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:28:38.391 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 144 = '' 2024-09-17T23:28:38.391 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:28:38.391 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.4 2024-09-17T23:28:38.792 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_30.status 2024-09-17T23:28:38.792 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:28:38.792 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:28:38.792 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_30 2024-09-17T23:28:38.793 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_30.status 2024-09-17T23:28:38.793 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2024-09-17T23:28:38.793 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:28:38.846 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:28:38.847 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:28:38.848 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:28:38.848 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:28:38.848 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:28:38.849 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:28:38.849 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 144 = '' 2024-09-17T23:28:38.849 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:28:38.849 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.8 2024-09-17T23:28:39.650 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_30.status 2024-09-17T23:28:39.650 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:28:39.651 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:28:39.651 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_30 2024-09-17T23:28:39.651 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_30.status 2024-09-17T23:28:39.651 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2024-09-17T23:28:39.651 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:28:39.698 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:28:39.700 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id=144 2024-09-17T23:28:39.700 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 144 = 144 2024-09-17T23:28:39.700 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:28:39.700 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snap_present cluster1 mirror image_30 snap 2024-09-17T23:28:39.700 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:28:39.701 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:28:39.701 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_30 2024-09-17T23:28:39.701 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:28:39.701 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:28:39.701 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-17T23:28:39.701 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:28:40.701 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 info mirror/image_30@snap 2024-09-17T23:28:40.751 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'image_30': 2024-09-17T23:28:40.751 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 128 MiB in 32 objects 2024-09-17T23:28:40.751 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:28:40.751 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 2 2024-09-17T23:28:40.751 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 1210542c3d6b 2024-09-17T23:28:40.752 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.1210542c3d6b 2024-09-17T23:28:40.752 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:28:40.752 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-17T23:28:40.752 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:28:40.752 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:28:40.752 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:21:21 2024 2024-09-17T23:28:40.752 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:21:21 2024 2024-09-17T23:28:40.752 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:21:21 2024 2024-09-17T23:28:40.752 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: protected: False 2024-09-17T23:28:40.752 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:28:40.752 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:28:40.752 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: c1039ee2-f5c2-417c-8ea6-8094c3f49b33 2024-09-17T23:28:40.753 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: false 2024-09-17T23:28:40.755 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:28:40.755 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ compare_image_snaps mirror image_30 snap 2024-09-17T23:28:40.755 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:28:40.755 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_30 2024-09-17T23:28:40.755 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:28:40.755 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local ret=0 2024-09-17T23:28:40.755 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local rmt_export=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_30.export 2024-09-17T23:28:40.755 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local loc_export=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_30.export 2024-09-17T23:28:40.755 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_30.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_30.export 2024-09-17T23:28:40.756 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror export image_30@snap /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_30.export 2024-09-17T23:28:40.931 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-17T23:28:40.939 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 -p mirror export image_30@snap /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_30.export 2024-09-17T23:28:41.154 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-17T23:28:41.161 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ cmp /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_30.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_30.export 2024-09-17T23:28:41.223 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_30.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_30.export 2024-09-17T23:28:41.255 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:28:41.255 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:28:41.255 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_31 2024-09-17T23:28:41.255 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_snap cluster2 mirror image_31 snap 2024-09-17T23:28:41.256 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:28:41.256 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:28:41.256 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_31 2024-09-17T23:28:41.256 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:28:41.256 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror snap create image_31@snap --debug-rbd=20 --debug-journaler=20 2024-09-17T23:28:43.098 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_started cluster1 mirror image_31 2024-09-17T23:28:43.098 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:28:43.099 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:28:43.099 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_31 2024-09-17T23:28:43.099 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_state cluster1 mirror image_31 started 2024-09-17T23:28:43.099 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:28:43.099 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:28:43.099 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_31 2024-09-17T23:28:43.099 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=started 2024-09-17T23:28:43.099 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:28:43.099 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:28:43.099 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:28:44.100 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_replay_state cluster1 mirror image_31 started 2024-09-17T23:28:44.100 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:28:44.100 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:28:44.100 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_31 2024-09-17T23:28:44.100 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=started 2024-09-17T23:28:44.100 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_result 2024-09-17T23:28:44.100 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=stopped 2024-09-17T23:28:44.101 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_31 2024-09-17T23:28:44.101 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_instance=cluster1 2024-09-17T23:28:44.101 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:28:44.101 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:28:44.101 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ grep -i state 2024-09-17T23:28:44.101 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance=cluster1 2024-09-17T23:28:44.102 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local loop_instance 2024-09-17T23:28:44.102 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:28:44.102 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sleep 0 2024-09-17T23:28:44.102 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-17T23:28:44.103 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ seq 0 3 2024-09-17T23:28:44.104 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:28:44.104 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_31 2024-09-17T23:28:44.104 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:0 2024-09-17T23:28:44.104 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:28:44.104 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:28:44.104 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-17T23:28:44.104 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:0 2024-09-17T23:28:44.104 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:28:44.104 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:28:44.104 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:28:44.104 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=0 2024-09-17T23:28:44.104 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-17T23:28:44.105 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:28:44.105 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:28:44.105 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=0 2024-09-17T23:28:44.105 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:28:44.105 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-17T23:28:44.105 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:0 2024-09-17T23:28:44.105 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:28:44.105 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:28:44.105 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-17T23:28:44.105 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:0 2024-09-17T23:28:44.105 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:28:44.106 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:28:44.106 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:28:44.106 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:28:44.106 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-17T23:28:44.107 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:28:44.107 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:28:44.107 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=0 2024-09-17T23:28:44.107 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=0 2024-09-17T23:28:44.107 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-17T23:28:44.126 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:28:44.126 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:28:44.126 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:28:44.126 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_31 2024-09-17T23:28:44.252 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ return 0 2024-09-17T23:28:44.253 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ status_result=' "state": "Replaying"' 2024-09-17T23:28:44.253 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ echo ' "state": "Replaying"' 2024-09-17T23:28:44.253 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep -i Replaying 2024-09-17T23:28:44.253 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: "state": "Replaying" 2024-09-17T23:28:44.254 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_state=started 2024-09-17T23:28:44.255 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test started = started 2024-09-17T23:28:44.255 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:28:44.255 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_31 2024-09-17T23:28:44.255 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:28:44.255 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:28:44.255 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:28:44.255 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_31 2024-09-17T23:28:44.255 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = journal ']' 2024-09-17T23:28:44.255 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = snapshot ']' 2024-09-17T23:28:44.255 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_31 2024-09-17T23:28:44.255 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:28:44.255 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:28:44.256 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:28:44.256 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_31 2024-09-17T23:28:44.256 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster2-mirror-image_31.status 2024-09-17T23:28:44.256 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster2-mirror-image_31.status 2024-09-17T23:28:44.256 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:28:44.256 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_31.status 2024-09-17T23:28:44.257 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster1-mirror-image_31.status 2024-09-17T23:28:44.257 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster1-mirror-image_31.status 2024-09-17T23:28:44.257 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:28:44.257 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_status_log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_31.status 2024-09-17T23:28:44.258 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ mirror_image_snapshot cluster2 mirror image_31 2024-09-17T23:28:44.258 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:28:44.258 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:28:44.258 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_31 2024-09-17T23:28:44.258 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_31 2024-09-17T23:28:45.106 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Snapshot ID: 146 2024-09-17T23:28:45.119 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_31 /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_31.status 2024-09-17T23:28:45.120 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:28:45.120 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:28:45.120 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_31 2024-09-17T23:28:45.120 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_31.status 2024-09-17T23:28:45.120 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_31 --format xml 2024-09-17T23:28:45.120 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:28:45.168 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-17T23:28:45.170 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snapshot_id=146 2024-09-17T23:28:45.170 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:28:45.170 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:28:45.170 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.2 2024-09-17T23:28:45.371 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_31.status 2024-09-17T23:28:45.372 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:28:45.372 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:28:45.372 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_31 2024-09-17T23:28:45.372 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_31.status 2024-09-17T23:28:45.372 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2024-09-17T23:28:45.372 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:28:45.627 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:28:45.627 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:28:45.629 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:28:45.629 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:28:45.629 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:28:45.630 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:28:45.630 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 146 = '' 2024-09-17T23:28:45.630 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:28:45.630 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.4 2024-09-17T23:28:46.031 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_31.status 2024-09-17T23:28:46.031 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:28:46.032 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:28:46.032 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_31 2024-09-17T23:28:46.032 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_31.status 2024-09-17T23:28:46.032 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2024-09-17T23:28:46.032 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:28:46.087 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:28:46.088 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:28:46.090 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:28:46.090 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:28:46.090 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:28:46.090 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:28:46.091 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 146 = '' 2024-09-17T23:28:46.091 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:28:46.091 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.8 2024-09-17T23:28:46.892 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_31.status 2024-09-17T23:28:46.892 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:28:46.893 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:28:46.893 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_31 2024-09-17T23:28:46.893 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_31.status 2024-09-17T23:28:46.893 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2024-09-17T23:28:46.893 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:28:46.941 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:28:46.944 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id=146 2024-09-17T23:28:46.944 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 146 = 146 2024-09-17T23:28:46.944 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:28:46.944 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snap_present cluster1 mirror image_31 snap 2024-09-17T23:28:46.944 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:28:46.944 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:28:46.944 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_31 2024-09-17T23:28:46.944 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:28:46.945 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:28:46.945 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-17T23:28:46.945 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:28:47.945 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 info mirror/image_31@snap 2024-09-17T23:28:47.993 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'image_31': 2024-09-17T23:28:47.993 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 128 MiB in 32 objects 2024-09-17T23:28:47.993 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:28:47.993 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 2 2024-09-17T23:28:47.994 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 1210e3574a97 2024-09-17T23:28:47.994 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.1210e3574a97 2024-09-17T23:28:47.994 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:28:47.994 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-17T23:28:47.994 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:28:47.994 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:28:47.994 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:21:33 2024 2024-09-17T23:28:47.994 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:21:33 2024 2024-09-17T23:28:47.994 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:21:33 2024 2024-09-17T23:28:47.994 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: protected: False 2024-09-17T23:28:47.994 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:28:47.994 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:28:47.994 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: c8ba8555-2fc3-405e-b6a5-7ff5d6a4334c 2024-09-17T23:28:47.994 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: false 2024-09-17T23:28:47.997 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:28:47.997 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ compare_image_snaps mirror image_31 snap 2024-09-17T23:28:47.997 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:28:47.998 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_31 2024-09-17T23:28:47.998 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:28:47.998 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local ret=0 2024-09-17T23:28:47.998 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local rmt_export=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_31.export 2024-09-17T23:28:47.998 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local loc_export=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_31.export 2024-09-17T23:28:47.998 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_31.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_31.export 2024-09-17T23:28:47.998 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror export image_31@snap /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_31.export 2024-09-17T23:28:48.169 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-17T23:28:48.177 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 -p mirror export image_31@snap /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_31.export 2024-09-17T23:28:48.396 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-17T23:28:48.402 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ cmp /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_31.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_31.export 2024-09-17T23:28:48.465 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_31.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_31.export 2024-09-17T23:28:48.497 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:28:48.498 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:28:48.498 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_32 2024-09-17T23:28:48.498 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_snap cluster2 mirror image_32 snap 2024-09-17T23:28:48.498 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:28:48.498 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:28:48.498 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_32 2024-09-17T23:28:48.498 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:28:48.498 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror snap create image_32@snap --debug-rbd=20 --debug-journaler=20 2024-09-17T23:28:50.126 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_started cluster1 mirror image_32 2024-09-17T23:28:50.126 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:28:50.126 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:28:50.126 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_32 2024-09-17T23:28:50.126 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_state cluster1 mirror image_32 started 2024-09-17T23:28:50.126 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:28:50.126 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:28:50.126 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_32 2024-09-17T23:28:50.126 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=started 2024-09-17T23:28:50.127 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:28:50.127 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:28:50.127 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:28:51.127 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_replay_state cluster1 mirror image_32 started 2024-09-17T23:28:51.127 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:28:51.127 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:28:51.127 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_32 2024-09-17T23:28:51.128 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=started 2024-09-17T23:28:51.128 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_result 2024-09-17T23:28:51.128 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=stopped 2024-09-17T23:28:51.129 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_32 2024-09-17T23:28:51.129 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_instance=cluster1 2024-09-17T23:28:51.129 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ grep -i state 2024-09-17T23:28:51.129 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:28:51.129 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:28:51.129 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance=cluster1 2024-09-17T23:28:51.129 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local loop_instance 2024-09-17T23:28:51.129 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:28:51.129 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sleep 0 2024-09-17T23:28:51.129 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-17T23:28:51.132 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ seq 0 3 2024-09-17T23:28:51.132 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:28:51.132 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_32 2024-09-17T23:28:51.132 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:0 2024-09-17T23:28:51.132 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:28:51.132 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:28:51.132 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-17T23:28:51.132 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:0 2024-09-17T23:28:51.132 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:28:51.132 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:28:51.132 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:28:51.132 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=0 2024-09-17T23:28:51.132 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-17T23:28:51.133 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:28:51.133 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:28:51.133 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=0 2024-09-17T23:28:51.133 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:28:51.133 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-17T23:28:51.133 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:0 2024-09-17T23:28:51.133 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:28:51.133 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:28:51.133 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-17T23:28:51.133 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:0 2024-09-17T23:28:51.133 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:28:51.133 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:28:51.133 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:28:51.134 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:28:51.134 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-17T23:28:51.134 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:28:51.134 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:28:51.134 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=0 2024-09-17T23:28:51.134 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=0 2024-09-17T23:28:51.134 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-17T23:28:51.154 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:28:51.154 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:28:51.154 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:28:51.154 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_32 2024-09-17T23:28:51.280 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ return 0 2024-09-17T23:28:51.281 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ status_result=' "state": "Replaying"' 2024-09-17T23:28:51.281 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ echo ' "state": "Replaying"' 2024-09-17T23:28:51.281 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep -i Replaying 2024-09-17T23:28:51.281 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: "state": "Replaying" 2024-09-17T23:28:51.283 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_state=started 2024-09-17T23:28:51.283 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test started = started 2024-09-17T23:28:51.283 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:28:51.283 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_32 2024-09-17T23:28:51.283 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:28:51.283 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:28:51.283 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:28:51.283 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_32 2024-09-17T23:28:51.283 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = journal ']' 2024-09-17T23:28:51.283 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = snapshot ']' 2024-09-17T23:28:51.284 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_32 2024-09-17T23:28:51.284 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:28:51.284 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:28:51.284 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:28:51.284 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_32 2024-09-17T23:28:51.284 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster2-mirror-image_32.status 2024-09-17T23:28:51.284 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster2-mirror-image_32.status 2024-09-17T23:28:51.284 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:28:51.285 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_32.status 2024-09-17T23:28:51.285 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster1-mirror-image_32.status 2024-09-17T23:28:51.285 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster1-mirror-image_32.status 2024-09-17T23:28:51.285 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:28:51.286 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_status_log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_32.status 2024-09-17T23:28:51.286 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ mirror_image_snapshot cluster2 mirror image_32 2024-09-17T23:28:51.286 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:28:51.286 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:28:51.286 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_32 2024-09-17T23:28:51.286 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_32 2024-09-17T23:28:52.137 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Snapshot ID: 148 2024-09-17T23:28:52.149 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_32 /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_32.status 2024-09-17T23:28:52.149 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:28:52.149 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:28:52.150 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_32 2024-09-17T23:28:52.150 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_32.status 2024-09-17T23:28:52.150 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_32 --format xml 2024-09-17T23:28:52.150 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:28:52.198 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-17T23:28:52.200 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snapshot_id=148 2024-09-17T23:28:52.200 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:28:52.200 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:28:52.200 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.2 2024-09-17T23:28:52.401 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_32.status 2024-09-17T23:28:52.402 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:28:52.402 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:28:52.402 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_32 2024-09-17T23:28:52.402 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_32.status 2024-09-17T23:28:52.402 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2024-09-17T23:28:52.402 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:28:52.451 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:28:52.451 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:28:52.453 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:28:52.453 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:28:52.453 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:28:52.453 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:28:52.454 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 148 = '' 2024-09-17T23:28:52.454 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:28:52.454 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.4 2024-09-17T23:28:52.855 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_32.status 2024-09-17T23:28:52.855 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:28:52.855 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:28:52.855 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_32 2024-09-17T23:28:52.855 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_32.status 2024-09-17T23:28:52.855 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2024-09-17T23:28:52.855 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:28:52.909 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:28:52.909 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:28:52.911 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:28:52.911 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:28:52.911 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:28:52.911 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:28:52.911 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 148 = '' 2024-09-17T23:28:52.912 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:28:52.912 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.8 2024-09-17T23:28:53.713 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_32.status 2024-09-17T23:28:53.713 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:28:53.713 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:28:53.713 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_32 2024-09-17T23:28:53.713 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_32.status 2024-09-17T23:28:53.713 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2024-09-17T23:28:53.714 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:28:53.759 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:28:53.761 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id=148 2024-09-17T23:28:53.761 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 148 = 148 2024-09-17T23:28:53.761 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:28:53.761 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snap_present cluster1 mirror image_32 snap 2024-09-17T23:28:53.761 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:28:53.762 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:28:53.762 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_32 2024-09-17T23:28:53.762 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:28:53.762 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:28:53.762 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-17T23:28:53.762 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:28:54.762 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 info mirror/image_32@snap 2024-09-17T23:28:54.810 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'image_32': 2024-09-17T23:28:54.810 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 128 MiB in 32 objects 2024-09-17T23:28:54.810 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:28:54.810 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 2 2024-09-17T23:28:54.810 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 121016b3d0f2 2024-09-17T23:28:54.810 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.121016b3d0f2 2024-09-17T23:28:54.810 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:28:54.810 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-17T23:28:54.810 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:28:54.810 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:28:54.810 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:21:45 2024 2024-09-17T23:28:54.811 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:21:45 2024 2024-09-17T23:28:54.811 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:21:45 2024 2024-09-17T23:28:54.811 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: protected: False 2024-09-17T23:28:54.811 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:28:54.811 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:28:54.811 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: c47737c4-4126-4b7d-8eff-21c600ab88e3 2024-09-17T23:28:54.811 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: false 2024-09-17T23:28:54.813 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:28:54.813 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ compare_image_snaps mirror image_32 snap 2024-09-17T23:28:54.814 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:28:54.814 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_32 2024-09-17T23:28:54.814 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:28:54.814 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local ret=0 2024-09-17T23:28:54.814 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local rmt_export=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_32.export 2024-09-17T23:28:54.814 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local loc_export=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_32.export 2024-09-17T23:28:54.814 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_32.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_32.export 2024-09-17T23:28:54.814 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror export image_32@snap /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_32.export 2024-09-17T23:28:54.983 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-17T23:28:54.990 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 -p mirror export image_32@snap /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_32.export 2024-09-17T23:28:55.231 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-17T23:28:55.239 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ cmp /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_32.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_32.export 2024-09-17T23:28:55.301 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_32.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_32.export 2024-09-17T23:28:55.333 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:28:55.333 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:28:55.334 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_33 2024-09-17T23:28:55.334 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_snap cluster2 mirror image_33 snap 2024-09-17T23:28:55.334 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:28:55.334 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:28:55.334 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_33 2024-09-17T23:28:55.334 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:28:55.334 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror snap create image_33@snap --debug-rbd=20 --debug-journaler=20 2024-09-17T23:28:57.159 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_started cluster1 mirror image_33 2024-09-17T23:28:57.159 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:28:57.159 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:28:57.160 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_33 2024-09-17T23:28:57.160 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_state cluster1 mirror image_33 started 2024-09-17T23:28:57.160 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:28:57.160 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:28:57.160 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_33 2024-09-17T23:28:57.160 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=started 2024-09-17T23:28:57.160 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:28:57.160 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:28:57.160 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:28:58.160 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_replay_state cluster1 mirror image_33 started 2024-09-17T23:28:58.161 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:28:58.161 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:28:58.161 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_33 2024-09-17T23:28:58.161 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=started 2024-09-17T23:28:58.161 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_result 2024-09-17T23:28:58.161 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=stopped 2024-09-17T23:28:58.162 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_33 2024-09-17T23:28:58.162 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_instance=cluster1 2024-09-17T23:28:58.162 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ grep -i state 2024-09-17T23:28:58.162 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:28:58.162 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:28:58.162 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance=cluster1 2024-09-17T23:28:58.163 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local loop_instance 2024-09-17T23:28:58.163 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:28:58.163 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sleep 0 2024-09-17T23:28:58.163 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-17T23:28:58.165 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ seq 0 3 2024-09-17T23:28:58.166 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:28:58.166 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_33 2024-09-17T23:28:58.166 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:0 2024-09-17T23:28:58.166 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:28:58.166 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:28:58.166 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-17T23:28:58.166 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:0 2024-09-17T23:28:58.166 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:28:58.166 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:28:58.166 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:28:58.166 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=0 2024-09-17T23:28:58.166 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-17T23:28:58.166 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:28:58.166 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:28:58.167 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=0 2024-09-17T23:28:58.167 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:28:58.167 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-17T23:28:58.167 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:0 2024-09-17T23:28:58.167 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:28:58.167 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:28:58.167 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-17T23:28:58.167 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:0 2024-09-17T23:28:58.167 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:28:58.167 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:28:58.167 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:28:58.167 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:28:58.167 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-17T23:28:58.167 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:28:58.168 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:28:58.168 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=0 2024-09-17T23:28:58.168 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=0 2024-09-17T23:28:58.168 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-17T23:28:58.187 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:28:58.188 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:28:58.188 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:28:58.188 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_33 2024-09-17T23:28:58.315 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ return 0 2024-09-17T23:28:58.316 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ status_result=' "state": "Replaying"' 2024-09-17T23:28:58.316 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ echo ' "state": "Replaying"' 2024-09-17T23:28:58.316 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep -i Replaying 2024-09-17T23:28:58.317 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: "state": "Replaying" 2024-09-17T23:28:58.318 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_state=started 2024-09-17T23:28:58.318 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test started = started 2024-09-17T23:28:58.318 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:28:58.318 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_33 2024-09-17T23:28:58.319 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:28:58.319 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:28:58.319 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:28:58.319 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_33 2024-09-17T23:28:58.319 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = journal ']' 2024-09-17T23:28:58.319 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = snapshot ']' 2024-09-17T23:28:58.319 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_33 2024-09-17T23:28:58.319 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:28:58.319 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:28:58.319 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:28:58.319 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_33 2024-09-17T23:28:58.319 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster2-mirror-image_33.status 2024-09-17T23:28:58.319 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster2-mirror-image_33.status 2024-09-17T23:28:58.320 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:28:58.320 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_33.status 2024-09-17T23:28:58.320 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster1-mirror-image_33.status 2024-09-17T23:28:58.320 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster1-mirror-image_33.status 2024-09-17T23:28:58.320 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:28:58.321 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_status_log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_33.status 2024-09-17T23:28:58.321 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ mirror_image_snapshot cluster2 mirror image_33 2024-09-17T23:28:58.321 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:28:58.321 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:28:58.322 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_33 2024-09-17T23:28:58.322 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_33 2024-09-17T23:28:59.174 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Snapshot ID: 150 2024-09-17T23:28:59.187 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_33 /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_33.status 2024-09-17T23:28:59.188 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:28:59.188 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:28:59.188 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_33 2024-09-17T23:28:59.188 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_33.status 2024-09-17T23:28:59.188 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_33 --format xml 2024-09-17T23:28:59.188 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:28:59.235 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-17T23:28:59.237 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snapshot_id=150 2024-09-17T23:28:59.237 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:28:59.237 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:28:59.237 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.2 2024-09-17T23:28:59.439 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_33 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_33.status 2024-09-17T23:28:59.439 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:28:59.439 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:28:59.439 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_33 2024-09-17T23:28:59.439 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_33.status 2024-09-17T23:28:59.439 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_33 --format xml 2024-09-17T23:28:59.439 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:28:59.492 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:28:59.492 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:28:59.493 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:28:59.493 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:28:59.493 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:28:59.493 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:28:59.493 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 150 = '' 2024-09-17T23:28:59.493 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:28:59.493 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.4 2024-09-17T23:28:59.893 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_33 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_33.status 2024-09-17T23:28:59.893 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:28:59.893 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:28:59.893 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_33 2024-09-17T23:28:59.893 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_33.status 2024-09-17T23:28:59.894 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_33 --format xml 2024-09-17T23:28:59.894 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:28:59.949 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:28:59.949 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:28:59.950 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:28:59.950 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:28:59.951 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:28:59.951 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:28:59.951 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 150 = '' 2024-09-17T23:28:59.952 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:28:59.952 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.8 2024-09-17T23:29:00.258 INFO:tasks.ceph.cluster2.osd.2.smithi038.stderr:2024-09-17T23:29:00.255+0000 7f1c4b04a640 -1 reset not still connected to 0x561cf092f860 2024-09-17T23:29:00.753 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_33 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_33.status 2024-09-17T23:29:00.753 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:29:00.753 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:29:00.753 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_33 2024-09-17T23:29:00.753 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_33.status 2024-09-17T23:29:00.753 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_33 --format xml 2024-09-17T23:29:00.753 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:29:00.801 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:29:00.803 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id=150 2024-09-17T23:29:00.803 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 150 = 150 2024-09-17T23:29:00.804 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:29:00.804 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snap_present cluster1 mirror image_33 snap 2024-09-17T23:29:00.804 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:29:00.804 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:29:00.804 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_33 2024-09-17T23:29:00.804 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:29:00.804 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:29:00.804 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-17T23:29:00.804 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:29:01.804 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 info mirror/image_33@snap 2024-09-17T23:29:01.853 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'image_33': 2024-09-17T23:29:01.853 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 128 MiB in 32 objects 2024-09-17T23:29:01.853 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:29:01.853 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 2 2024-09-17T23:29:01.853 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 12109a19dcd 2024-09-17T23:29:01.853 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.12109a19dcd 2024-09-17T23:29:01.853 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:29:01.853 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-17T23:29:01.854 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:29:01.854 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:29:01.854 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:21:55 2024 2024-09-17T23:29:01.854 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:21:55 2024 2024-09-17T23:29:01.854 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:21:55 2024 2024-09-17T23:29:01.854 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: protected: False 2024-09-17T23:29:01.854 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:29:01.854 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:29:01.854 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: d3b01291-4a3c-45c8-9b7e-71a70aa71cd3 2024-09-17T23:29:01.854 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: false 2024-09-17T23:29:01.857 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:29:01.857 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ compare_image_snaps mirror image_33 snap 2024-09-17T23:29:01.857 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:29:01.857 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_33 2024-09-17T23:29:01.857 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:29:01.857 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local ret=0 2024-09-17T23:29:01.857 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local rmt_export=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_33.export 2024-09-17T23:29:01.857 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local loc_export=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_33.export 2024-09-17T23:29:01.857 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_33.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_33.export 2024-09-17T23:29:01.858 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror export image_33@snap /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_33.export 2024-09-17T23:29:02.018 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-17T23:29:02.025 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 -p mirror export image_33@snap /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_33.export 2024-09-17T23:29:02.254 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-17T23:29:02.262 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ cmp /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_33.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_33.export 2024-09-17T23:29:02.325 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_33.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_33.export 2024-09-17T23:29:02.357 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:29:02.357 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:29:02.358 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_34 2024-09-17T23:29:02.358 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_snap cluster2 mirror image_34 snap 2024-09-17T23:29:02.358 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:29:02.358 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:29:02.358 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_34 2024-09-17T23:29:02.358 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:29:02.358 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror snap create image_34@snap --debug-rbd=20 --debug-journaler=20 2024-09-17T23:29:04.204 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_started cluster1 mirror image_34 2024-09-17T23:29:04.205 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:29:04.205 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:29:04.205 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_34 2024-09-17T23:29:04.205 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_state cluster1 mirror image_34 started 2024-09-17T23:29:04.205 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:29:04.205 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:29:04.205 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_34 2024-09-17T23:29:04.205 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=started 2024-09-17T23:29:04.205 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:29:04.205 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:29:04.205 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:29:04.955 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-09-17T23:29:04.955 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-09-17T23:29:04.956 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2024-09-17T23:29:04.956 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi038.stderr:2024-09-17T23:29:04.954+0000 7fdb2b2d2640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 49266) UID: 1000 2024-09-17T23:29:05.206 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_replay_state cluster1 mirror image_34 started 2024-09-17T23:29:05.206 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:29:05.206 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:29:05.206 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_34 2024-09-17T23:29:05.206 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=started 2024-09-17T23:29:05.206 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_result 2024-09-17T23:29:05.206 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=stopped 2024-09-17T23:29:05.207 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_34 2024-09-17T23:29:05.207 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ grep -i state 2024-09-17T23:29:05.207 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_instance=cluster1 2024-09-17T23:29:05.208 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:29:05.208 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:29:05.208 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance=cluster1 2024-09-17T23:29:05.208 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local loop_instance 2024-09-17T23:29:05.208 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:29:05.208 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sleep 0 2024-09-17T23:29:05.208 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-17T23:29:05.211 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ seq 0 3 2024-09-17T23:29:05.211 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:29:05.211 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_34 2024-09-17T23:29:05.211 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:0 2024-09-17T23:29:05.211 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:29:05.211 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:29:05.211 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-17T23:29:05.211 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:0 2024-09-17T23:29:05.211 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:29:05.212 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:29:05.212 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:29:05.212 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=0 2024-09-17T23:29:05.212 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-17T23:29:05.212 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:29:05.212 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:29:05.212 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=0 2024-09-17T23:29:05.212 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:29:05.212 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-17T23:29:05.212 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:0 2024-09-17T23:29:05.212 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:29:05.212 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:29:05.212 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-17T23:29:05.212 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:0 2024-09-17T23:29:05.213 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:29:05.213 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:29:05.213 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:29:05.213 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:29:05.213 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-17T23:29:05.213 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:29:05.213 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:29:05.213 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=0 2024-09-17T23:29:05.213 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=0 2024-09-17T23:29:05.213 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-17T23:29:05.232 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:29:05.233 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:29:05.233 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:29:05.233 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_34 2024-09-17T23:29:05.360 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ return 0 2024-09-17T23:29:05.361 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ status_result=' "state": "Replaying"' 2024-09-17T23:29:05.361 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ echo ' "state": "Replaying"' 2024-09-17T23:29:05.361 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep -i Replaying 2024-09-17T23:29:05.362 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: "state": "Replaying" 2024-09-17T23:29:05.363 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_state=started 2024-09-17T23:29:05.363 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test started = started 2024-09-17T23:29:05.363 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:29:05.363 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_34 2024-09-17T23:29:05.363 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:29:05.364 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:29:05.364 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:29:05.364 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_34 2024-09-17T23:29:05.364 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = journal ']' 2024-09-17T23:29:05.364 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = snapshot ']' 2024-09-17T23:29:05.364 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_34 2024-09-17T23:29:05.364 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:29:05.364 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:29:05.364 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:29:05.364 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_34 2024-09-17T23:29:05.364 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster2-mirror-image_34.status 2024-09-17T23:29:05.364 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster2-mirror-image_34.status 2024-09-17T23:29:05.364 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:29:05.365 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_34.status 2024-09-17T23:29:05.365 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster1-mirror-image_34.status 2024-09-17T23:29:05.365 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster1-mirror-image_34.status 2024-09-17T23:29:05.365 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:29:05.366 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_status_log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_34.status 2024-09-17T23:29:05.366 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ mirror_image_snapshot cluster2 mirror image_34 2024-09-17T23:29:05.366 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:29:05.366 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:29:05.366 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_34 2024-09-17T23:29:05.366 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_34 2024-09-17T23:29:06.215 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Snapshot ID: 152 2024-09-17T23:29:06.228 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_34 /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_34.status 2024-09-17T23:29:06.228 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:29:06.228 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:29:06.228 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_34 2024-09-17T23:29:06.228 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_34.status 2024-09-17T23:29:06.229 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_34 --format xml 2024-09-17T23:29:06.229 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:29:06.276 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-17T23:29:06.278 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snapshot_id=152 2024-09-17T23:29:06.278 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:29:06.278 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:29:06.278 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.2 2024-09-17T23:29:06.479 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_34.status 2024-09-17T23:29:06.479 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:29:06.480 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:29:06.480 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_34 2024-09-17T23:29:06.480 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_34.status 2024-09-17T23:29:06.480 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2024-09-17T23:29:06.480 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:29:06.532 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:29:06.533 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:29:06.534 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:29:06.534 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:29:06.534 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:29:06.535 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:29:06.535 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 152 = '' 2024-09-17T23:29:06.535 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:29:06.535 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.4 2024-09-17T23:29:06.936 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_34.status 2024-09-17T23:29:06.936 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:29:06.937 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:29:06.937 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_34 2024-09-17T23:29:06.937 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_34.status 2024-09-17T23:29:06.937 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2024-09-17T23:29:06.937 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:29:06.990 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:29:06.990 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:29:06.991 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:29:06.991 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:29:06.991 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:29:06.992 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:29:06.992 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 152 = '' 2024-09-17T23:29:06.992 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:29:06.992 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.8 2024-09-17T23:29:07.793 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_34.status 2024-09-17T23:29:07.794 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:29:07.794 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:29:07.794 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_34 2024-09-17T23:29:07.794 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_34.status 2024-09-17T23:29:07.794 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2024-09-17T23:29:07.794 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:29:07.840 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:29:07.842 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id=152 2024-09-17T23:29:07.842 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 152 = 152 2024-09-17T23:29:07.842 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:29:07.842 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snap_present cluster1 mirror image_34 snap 2024-09-17T23:29:07.843 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:29:07.843 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:29:07.843 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_34 2024-09-17T23:29:07.843 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:29:07.843 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:29:07.843 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-17T23:29:07.843 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:29:08.843 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 info mirror/image_34@snap 2024-09-17T23:29:08.891 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'image_34': 2024-09-17T23:29:08.892 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 128 MiB in 32 objects 2024-09-17T23:29:08.892 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:29:08.892 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 2 2024-09-17T23:29:08.892 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 12103cc95a79 2024-09-17T23:29:08.892 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.12103cc95a79 2024-09-17T23:29:08.892 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:29:08.892 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-17T23:29:08.892 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:29:08.892 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:29:08.892 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:22:10 2024 2024-09-17T23:29:08.892 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:22:10 2024 2024-09-17T23:29:08.893 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:22:10 2024 2024-09-17T23:29:08.893 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: protected: False 2024-09-17T23:29:08.893 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:29:08.893 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:29:08.893 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: 78b7bbf1-8bd9-4fb6-b1bb-58a0b0412b92 2024-09-17T23:29:08.893 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: false 2024-09-17T23:29:08.896 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:29:08.896 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ compare_image_snaps mirror image_34 snap 2024-09-17T23:29:08.896 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:29:08.896 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_34 2024-09-17T23:29:08.896 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:29:08.896 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local ret=0 2024-09-17T23:29:08.896 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local rmt_export=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_34.export 2024-09-17T23:29:08.896 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local loc_export=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_34.export 2024-09-17T23:29:08.896 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_34.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_34.export 2024-09-17T23:29:08.897 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror export image_34@snap /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_34.export 2024-09-17T23:29:09.069 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-17T23:29:09.077 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 -p mirror export image_34@snap /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_34.export 2024-09-17T23:29:09.299 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-17T23:29:09.307 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ cmp /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_34.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_34.export 2024-09-17T23:29:09.369 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_34.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_34.export 2024-09-17T23:29:09.402 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:29:09.403 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:29:09.403 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_35 2024-09-17T23:29:09.403 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_snap cluster2 mirror image_35 snap 2024-09-17T23:29:09.403 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:29:09.403 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:29:09.403 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_35 2024-09-17T23:29:09.403 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:29:09.403 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror snap create image_35@snap --debug-rbd=20 --debug-journaler=20 2024-09-17T23:29:11.258 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_started cluster1 mirror image_35 2024-09-17T23:29:11.259 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:29:11.259 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:29:11.259 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_35 2024-09-17T23:29:11.259 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_state cluster1 mirror image_35 started 2024-09-17T23:29:11.259 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:29:11.259 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:29:11.259 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_35 2024-09-17T23:29:11.259 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=started 2024-09-17T23:29:11.259 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:29:11.259 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:29:11.259 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:29:11.956 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-09-17T23:29:11.957 INFO:teuthology.orchestra.run:waiting for 600 2024-09-17T23:29:11.957 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-09-17T23:29:11.957 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-09-17T23:29:11.957 DEBUG:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-09-17T23:29:11.961 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-09-17T23:29:11.961 INFO:tasks.rbd_mirror_thrash:waiting for 95 secs before thrashing 2024-09-17T23:29:12.259 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_replay_state cluster1 mirror image_35 started 2024-09-17T23:29:12.259 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:29:12.259 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:29:12.260 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_35 2024-09-17T23:29:12.260 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=started 2024-09-17T23:29:12.260 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_result 2024-09-17T23:29:12.260 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=stopped 2024-09-17T23:29:12.261 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_35 2024-09-17T23:29:12.261 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_instance=cluster1 2024-09-17T23:29:12.261 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:29:12.261 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ grep -i state 2024-09-17T23:29:12.261 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:29:12.261 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance=cluster1 2024-09-17T23:29:12.261 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local loop_instance 2024-09-17T23:29:12.262 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:29:12.262 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sleep 0 2024-09-17T23:29:12.262 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-17T23:29:12.262 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ seq 0 3 2024-09-17T23:29:12.263 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:29:12.263 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_35 2024-09-17T23:29:12.264 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:0 2024-09-17T23:29:12.264 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:29:12.264 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:29:12.264 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-17T23:29:12.264 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:0 2024-09-17T23:29:12.264 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:29:12.264 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:29:12.264 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:29:12.264 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=0 2024-09-17T23:29:12.264 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-17T23:29:12.264 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:29:12.264 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:29:12.264 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=0 2024-09-17T23:29:12.265 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:29:12.265 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-17T23:29:12.266 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:0 2024-09-17T23:29:12.266 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:29:12.266 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:29:12.266 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-17T23:29:12.266 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:0 2024-09-17T23:29:12.266 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:29:12.267 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:29:12.267 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:29:12.267 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:29:12.267 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-17T23:29:12.267 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:29:12.267 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:29:12.267 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=0 2024-09-17T23:29:12.267 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=0 2024-09-17T23:29:12.267 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-17T23:29:12.282 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:29:12.282 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:29:12.282 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:29:12.283 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_35 2024-09-17T23:29:12.403 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ return 0 2024-09-17T23:29:12.404 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ status_result=' "state": "Replaying"' 2024-09-17T23:29:12.404 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ echo ' "state": "Replaying"' 2024-09-17T23:29:12.404 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep -i Replaying 2024-09-17T23:29:12.405 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: "state": "Replaying" 2024-09-17T23:29:12.406 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_state=started 2024-09-17T23:29:12.407 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test started = started 2024-09-17T23:29:12.407 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:29:12.407 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_35 2024-09-17T23:29:12.407 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:29:12.407 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:29:12.407 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:29:12.407 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_35 2024-09-17T23:29:12.407 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = journal ']' 2024-09-17T23:29:12.407 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = snapshot ']' 2024-09-17T23:29:12.408 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_35 2024-09-17T23:29:12.408 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:29:12.408 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:29:12.408 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:29:12.408 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_35 2024-09-17T23:29:12.408 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster2-mirror-image_35.status 2024-09-17T23:29:12.408 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster2-mirror-image_35.status 2024-09-17T23:29:12.408 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:29:12.408 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_35.status 2024-09-17T23:29:12.409 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster1-mirror-image_35.status 2024-09-17T23:29:12.409 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster1-mirror-image_35.status 2024-09-17T23:29:12.410 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:29:12.410 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_status_log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_35.status 2024-09-17T23:29:12.410 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ mirror_image_snapshot cluster2 mirror image_35 2024-09-17T23:29:12.410 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:29:12.410 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:29:12.410 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_35 2024-09-17T23:29:12.410 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_35 2024-09-17T23:29:13.254 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Snapshot ID: 154 2024-09-17T23:29:13.266 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_35 /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_35.status 2024-09-17T23:29:13.266 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:29:13.267 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:29:13.267 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_35 2024-09-17T23:29:13.267 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_35.status 2024-09-17T23:29:13.267 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_35 --format xml 2024-09-17T23:29:13.267 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:29:13.314 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-17T23:29:13.316 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snapshot_id=154 2024-09-17T23:29:13.316 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:29:13.316 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:29:13.316 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.2 2024-09-17T23:29:13.517 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_35 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_35.status 2024-09-17T23:29:13.517 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:29:13.518 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:29:13.518 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_35 2024-09-17T23:29:13.518 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_35.status 2024-09-17T23:29:13.518 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_35 --format xml 2024-09-17T23:29:13.518 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:29:13.569 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:29:13.571 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id=69 2024-09-17T23:29:13.571 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 154 = 69 2024-09-17T23:29:13.571 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:29:13.571 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.4 2024-09-17T23:29:13.972 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_35 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_35.status 2024-09-17T23:29:13.973 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:29:13.973 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:29:13.973 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_35 2024-09-17T23:29:13.973 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_35.status 2024-09-17T23:29:13.973 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_35 --format xml 2024-09-17T23:29:13.973 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:29:14.027 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:29:14.029 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id=69 2024-09-17T23:29:14.029 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 154 = 69 2024-09-17T23:29:14.029 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:29:14.029 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.8 2024-09-17T23:29:14.830 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_35 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_35.status 2024-09-17T23:29:14.830 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:29:14.831 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:29:14.831 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_35 2024-09-17T23:29:14.831 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_35.status 2024-09-17T23:29:14.831 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_35 --format xml 2024-09-17T23:29:14.831 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:29:15.085 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:29:15.086 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:29:15.087 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:29:15.087 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:29:15.087 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:29:15.088 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:29:15.088 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 154 = '' 2024-09-17T23:29:15.088 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:29:15.088 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1.6 2024-09-17T23:29:16.689 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_35 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_35.status 2024-09-17T23:29:16.690 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:29:16.690 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:29:16.690 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_35 2024-09-17T23:29:16.690 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_35.status 2024-09-17T23:29:16.690 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_35 --format xml 2024-09-17T23:29:16.690 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:29:16.746 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:29:16.749 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id=154 2024-09-17T23:29:16.749 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 154 = 154 2024-09-17T23:29:16.749 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:29:16.749 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snap_present cluster1 mirror image_35 snap 2024-09-17T23:29:16.749 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:29:16.749 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:29:16.749 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_35 2024-09-17T23:29:16.749 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:29:16.750 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:29:16.750 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-17T23:29:16.750 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:29:17.750 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 info mirror/image_35@snap 2024-09-17T23:29:17.799 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'image_35': 2024-09-17T23:29:17.800 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 128 MiB in 32 objects 2024-09-17T23:29:17.800 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:29:17.800 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 2 2024-09-17T23:29:17.800 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 1210ece0dd0c 2024-09-17T23:29:17.800 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.1210ece0dd0c 2024-09-17T23:29:17.800 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:29:17.800 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-17T23:29:17.800 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:29:17.800 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:29:17.800 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:22:17 2024 2024-09-17T23:29:17.800 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:22:17 2024 2024-09-17T23:29:17.800 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:22:17 2024 2024-09-17T23:29:17.801 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: protected: False 2024-09-17T23:29:17.801 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:29:17.801 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:29:17.801 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: b8f54ad1-6d1a-42f7-821f-2ff1a07ca075 2024-09-17T23:29:17.801 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: false 2024-09-17T23:29:17.803 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:29:17.803 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ compare_image_snaps mirror image_35 snap 2024-09-17T23:29:17.804 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:29:17.804 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_35 2024-09-17T23:29:17.804 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:29:17.804 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local ret=0 2024-09-17T23:29:17.804 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local rmt_export=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_35.export 2024-09-17T23:29:17.804 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local loc_export=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_35.export 2024-09-17T23:29:17.804 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_35.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_35.export 2024-09-17T23:29:17.804 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror export image_35@snap /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_35.export 2024-09-17T23:29:17.973 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-17T23:29:17.980 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 -p mirror export image_35@snap /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_35.export 2024-09-17T23:29:18.171 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-17T23:29:18.179 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ cmp /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_35.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_35.export 2024-09-17T23:29:18.242 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_35.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_35.export 2024-09-17T23:29:18.275 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:29:18.275 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:29:18.275 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_36 2024-09-17T23:29:18.275 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_snap cluster2 mirror image_36 snap 2024-09-17T23:29:18.275 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:29:18.275 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:29:18.275 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_36 2024-09-17T23:29:18.276 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:29:18.276 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror snap create image_36@snap --debug-rbd=20 --debug-journaler=20 2024-09-17T23:29:20.281 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_started cluster1 mirror image_36 2024-09-17T23:29:20.281 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:29:20.281 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:29:20.281 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_36 2024-09-17T23:29:20.281 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_state cluster1 mirror image_36 started 2024-09-17T23:29:20.281 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:29:20.282 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:29:20.282 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_36 2024-09-17T23:29:20.282 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=started 2024-09-17T23:29:20.282 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:29:20.282 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:29:20.282 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:29:21.282 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_replay_state cluster1 mirror image_36 started 2024-09-17T23:29:21.283 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:29:21.283 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:29:21.283 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_36 2024-09-17T23:29:21.283 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=started 2024-09-17T23:29:21.283 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_result 2024-09-17T23:29:21.283 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=stopped 2024-09-17T23:29:21.284 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_36 2024-09-17T23:29:21.284 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_instance=cluster1 2024-09-17T23:29:21.284 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ grep -i state 2024-09-17T23:29:21.284 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:29:21.284 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:29:21.284 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance=cluster1 2024-09-17T23:29:21.284 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local loop_instance 2024-09-17T23:29:21.284 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:29:21.285 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sleep 0 2024-09-17T23:29:21.285 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-17T23:29:21.286 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ seq 0 3 2024-09-17T23:29:21.287 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:29:21.287 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_36 2024-09-17T23:29:21.287 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:0 2024-09-17T23:29:21.287 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:29:21.287 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:29:21.287 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-17T23:29:21.287 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:0 2024-09-17T23:29:21.287 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:29:21.287 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:29:21.287 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:29:21.287 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=0 2024-09-17T23:29:21.287 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-17T23:29:21.287 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:29:21.287 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:29:21.288 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=0 2024-09-17T23:29:21.288 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:29:21.288 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-17T23:29:21.288 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:0 2024-09-17T23:29:21.288 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:29:21.288 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:29:21.288 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-17T23:29:21.288 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:0 2024-09-17T23:29:21.288 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:29:21.288 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:29:21.288 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:29:21.289 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:29:21.289 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-17T23:29:21.289 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:29:21.289 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:29:21.289 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=0 2024-09-17T23:29:21.289 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=0 2024-09-17T23:29:21.289 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-17T23:29:21.309 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:29:21.309 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:29:21.310 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:29:21.310 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_36 2024-09-17T23:29:21.437 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ return 0 2024-09-17T23:29:21.438 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ status_result=' "state": "Replaying"' 2024-09-17T23:29:21.438 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ echo ' "state": "Replaying"' 2024-09-17T23:29:21.438 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep -i Replaying 2024-09-17T23:29:21.439 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: "state": "Replaying" 2024-09-17T23:29:21.440 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_state=started 2024-09-17T23:29:21.440 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test started = started 2024-09-17T23:29:21.440 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:29:21.440 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_36 2024-09-17T23:29:21.441 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:29:21.441 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:29:21.441 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:29:21.441 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_36 2024-09-17T23:29:21.441 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = journal ']' 2024-09-17T23:29:21.441 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = snapshot ']' 2024-09-17T23:29:21.441 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_36 2024-09-17T23:29:21.441 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:29:21.441 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:29:21.441 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:29:21.441 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_36 2024-09-17T23:29:21.441 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster2-mirror-image_36.status 2024-09-17T23:29:21.441 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster2-mirror-image_36.status 2024-09-17T23:29:21.442 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:29:21.442 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_36.status 2024-09-17T23:29:21.442 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster1-mirror-image_36.status 2024-09-17T23:29:21.442 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster1-mirror-image_36.status 2024-09-17T23:29:21.442 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:29:21.443 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_status_log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_36.status 2024-09-17T23:29:21.443 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ mirror_image_snapshot cluster2 mirror image_36 2024-09-17T23:29:21.443 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:29:21.443 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:29:21.444 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_36 2024-09-17T23:29:21.444 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_36 2024-09-17T23:29:22.281 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Snapshot ID: 156 2024-09-17T23:29:22.294 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_36 /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_36.status 2024-09-17T23:29:22.294 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:29:22.294 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:29:22.294 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_36 2024-09-17T23:29:22.294 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_36.status 2024-09-17T23:29:22.295 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_36 --format xml 2024-09-17T23:29:22.295 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:29:22.341 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-17T23:29:22.343 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snapshot_id=156 2024-09-17T23:29:22.343 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:29:22.343 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:29:22.343 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.2 2024-09-17T23:29:22.544 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_36 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_36.status 2024-09-17T23:29:22.545 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:29:22.545 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:29:22.545 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_36 2024-09-17T23:29:22.545 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_36.status 2024-09-17T23:29:22.545 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_36 --format xml 2024-09-17T23:29:22.545 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:29:22.598 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:29:22.600 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id=70 2024-09-17T23:29:22.600 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 156 = 70 2024-09-17T23:29:22.600 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:29:22.600 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.4 2024-09-17T23:29:23.002 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_36 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_36.status 2024-09-17T23:29:23.002 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:29:23.002 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:29:23.002 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_36 2024-09-17T23:29:23.002 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_36.status 2024-09-17T23:29:23.002 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_36 --format xml 2024-09-17T23:29:23.002 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:29:23.057 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:29:23.059 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id=70 2024-09-17T23:29:23.059 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 156 = 70 2024-09-17T23:29:23.059 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:29:23.060 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.8 2024-09-17T23:29:23.861 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_36 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_36.status 2024-09-17T23:29:23.861 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:29:23.861 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:29:23.861 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_36 2024-09-17T23:29:23.861 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_36.status 2024-09-17T23:29:23.862 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_36 --format xml 2024-09-17T23:29:23.862 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:29:23.915 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:29:23.915 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:29:23.916 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:29:23.917 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:29:23.917 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:29:23.917 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:29:23.917 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 156 = '' 2024-09-17T23:29:23.917 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:29:23.918 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1.6 2024-09-17T23:29:25.519 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_36 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_36.status 2024-09-17T23:29:25.519 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:29:25.519 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:29:25.519 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_36 2024-09-17T23:29:25.519 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_36.status 2024-09-17T23:29:25.519 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_36 --format xml 2024-09-17T23:29:25.519 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:29:25.573 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:29:25.575 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id=156 2024-09-17T23:29:25.575 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 156 = 156 2024-09-17T23:29:25.575 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:29:25.576 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snap_present cluster1 mirror image_36 snap 2024-09-17T23:29:25.576 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:29:25.576 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:29:25.576 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_36 2024-09-17T23:29:25.576 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:29:25.576 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:29:25.576 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-17T23:29:25.576 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:29:26.577 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 info mirror/image_36@snap 2024-09-17T23:29:26.626 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'image_36': 2024-09-17T23:29:26.626 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 128 MiB in 32 objects 2024-09-17T23:29:26.626 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:29:26.626 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 2 2024-09-17T23:29:26.626 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 12102d883ad4 2024-09-17T23:29:26.626 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.12102d883ad4 2024-09-17T23:29:26.626 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:29:26.626 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-17T23:29:26.626 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:29:26.626 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:29:26.626 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:22:29 2024 2024-09-17T23:29:26.627 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:22:29 2024 2024-09-17T23:29:26.627 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:22:29 2024 2024-09-17T23:29:26.627 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: protected: False 2024-09-17T23:29:26.627 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:29:26.627 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:29:26.627 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: 2e12bc1d-2eb3-49b1-b5a5-e5c5013f32d1 2024-09-17T23:29:26.627 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: false 2024-09-17T23:29:26.630 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:29:26.630 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ compare_image_snaps mirror image_36 snap 2024-09-17T23:29:26.630 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:29:26.630 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_36 2024-09-17T23:29:26.630 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:29:26.630 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local ret=0 2024-09-17T23:29:26.630 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local rmt_export=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_36.export 2024-09-17T23:29:26.630 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local loc_export=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_36.export 2024-09-17T23:29:26.630 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_36.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_36.export 2024-09-17T23:29:26.631 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror export image_36@snap /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_36.export 2024-09-17T23:29:26.804 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-17T23:29:26.810 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 -p mirror export image_36@snap /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_36.export 2024-09-17T23:29:27.050 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-17T23:29:27.057 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ cmp /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_36.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_36.export 2024-09-17T23:29:27.121 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_36.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_36.export 2024-09-17T23:29:27.154 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:29:27.154 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:29:27.154 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_37 2024-09-17T23:29:27.154 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_snap cluster2 mirror image_37 snap 2024-09-17T23:29:27.154 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:29:27.154 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:29:27.154 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_37 2024-09-17T23:29:27.154 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:29:27.154 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror snap create image_37@snap --debug-rbd=20 --debug-journaler=20 2024-09-17T23:29:29.004 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_started cluster1 mirror image_37 2024-09-17T23:29:29.004 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:29:29.004 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:29:29.004 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_37 2024-09-17T23:29:29.004 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_state cluster1 mirror image_37 started 2024-09-17T23:29:29.005 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:29:29.005 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:29:29.005 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_37 2024-09-17T23:29:29.005 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=started 2024-09-17T23:29:29.005 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:29:29.005 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:29:29.005 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:29:30.012 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_replay_state cluster1 mirror image_37 started 2024-09-17T23:29:30.012 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:29:30.012 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:29:30.012 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_37 2024-09-17T23:29:30.012 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=started 2024-09-17T23:29:30.013 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_result 2024-09-17T23:29:30.013 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=stopped 2024-09-17T23:29:30.013 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_37 2024-09-17T23:29:30.013 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_instance=cluster1 2024-09-17T23:29:30.013 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ grep -i state 2024-09-17T23:29:30.013 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:29:30.013 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:29:30.013 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance=cluster1 2024-09-17T23:29:30.013 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local loop_instance 2024-09-17T23:29:30.013 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:29:30.013 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sleep 0 2024-09-17T23:29:30.013 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-17T23:29:30.014 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ seq 0 3 2024-09-17T23:29:30.014 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:29:30.014 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_37 2024-09-17T23:29:30.014 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:0 2024-09-17T23:29:30.014 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:29:30.014 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:29:30.014 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-17T23:29:30.014 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:0 2024-09-17T23:29:30.014 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:29:30.014 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:29:30.014 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:29:30.014 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=0 2024-09-17T23:29:30.014 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-17T23:29:30.015 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:29:30.015 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:29:30.015 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=0 2024-09-17T23:29:30.015 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:29:30.015 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-17T23:29:30.015 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:0 2024-09-17T23:29:30.015 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:29:30.015 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:29:30.015 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-17T23:29:30.015 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:0 2024-09-17T23:29:30.015 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:29:30.015 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:29:30.015 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:29:30.016 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:29:30.016 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-17T23:29:30.016 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:29:30.016 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:29:30.016 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=0 2024-09-17T23:29:30.016 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=0 2024-09-17T23:29:30.016 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-17T23:29:30.033 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:29:30.033 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:29:30.033 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:29:30.033 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_37 2024-09-17T23:29:30.161 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ return 0 2024-09-17T23:29:30.161 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ status_result=' "state": "Replaying"' 2024-09-17T23:29:30.161 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ echo ' "state": "Replaying"' 2024-09-17T23:29:30.161 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep -i Replaying 2024-09-17T23:29:30.162 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: "state": "Replaying" 2024-09-17T23:29:30.163 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_state=started 2024-09-17T23:29:30.163 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test started = started 2024-09-17T23:29:30.164 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:29:30.164 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_37 2024-09-17T23:29:30.164 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:29:30.164 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:29:30.164 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:29:30.164 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_37 2024-09-17T23:29:30.164 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = journal ']' 2024-09-17T23:29:30.164 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = snapshot ']' 2024-09-17T23:29:30.164 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_37 2024-09-17T23:29:30.164 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:29:30.164 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:29:30.164 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:29:30.164 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_37 2024-09-17T23:29:30.165 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster2-mirror-image_37.status 2024-09-17T23:29:30.165 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster2-mirror-image_37.status 2024-09-17T23:29:30.165 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:29:30.166 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_37.status 2024-09-17T23:29:30.166 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster1-mirror-image_37.status 2024-09-17T23:29:30.166 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster1-mirror-image_37.status 2024-09-17T23:29:30.166 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:29:30.166 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_status_log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_37.status 2024-09-17T23:29:30.166 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ mirror_image_snapshot cluster2 mirror image_37 2024-09-17T23:29:30.166 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:29:30.166 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:29:30.166 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_37 2024-09-17T23:29:30.167 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_37 2024-09-17T23:29:31.021 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Snapshot ID: 158 2024-09-17T23:29:31.036 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_37 /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_37.status 2024-09-17T23:29:31.036 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:29:31.037 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:29:31.037 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_37 2024-09-17T23:29:31.037 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_37.status 2024-09-17T23:29:31.037 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_37 --format xml 2024-09-17T23:29:31.037 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:29:31.085 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-17T23:29:31.087 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snapshot_id=158 2024-09-17T23:29:31.087 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:29:31.087 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:29:31.087 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.2 2024-09-17T23:29:31.289 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_37 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_37.status 2024-09-17T23:29:31.289 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:29:31.289 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:29:31.289 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_37 2024-09-17T23:29:31.290 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_37.status 2024-09-17T23:29:31.290 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_37 --format xml 2024-09-17T23:29:31.290 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:29:31.340 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:29:31.340 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:29:31.341 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:29:31.341 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:29:31.342 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:29:31.342 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:29:31.342 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 158 = '' 2024-09-17T23:29:31.343 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:29:31.343 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.4 2024-09-17T23:29:31.743 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_37 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_37.status 2024-09-17T23:29:31.744 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:29:31.744 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:29:31.744 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_37 2024-09-17T23:29:31.744 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_37.status 2024-09-17T23:29:31.744 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_37 --format xml 2024-09-17T23:29:31.744 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:29:31.796 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:29:31.796 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:29:31.798 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:29:31.798 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:29:31.798 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:29:31.799 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:29:31.799 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 158 = '' 2024-09-17T23:29:31.799 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:29:31.799 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.8 2024-09-17T23:29:32.600 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_37 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_37.status 2024-09-17T23:29:32.600 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:29:32.600 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:29:32.600 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_37 2024-09-17T23:29:32.600 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_37.status 2024-09-17T23:29:32.600 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_37 --format xml 2024-09-17T23:29:32.601 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:29:32.646 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:29:32.648 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id=158 2024-09-17T23:29:32.648 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 158 = 158 2024-09-17T23:29:32.648 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:29:32.648 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snap_present cluster1 mirror image_37 snap 2024-09-17T23:29:32.648 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:29:32.649 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:29:32.649 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_37 2024-09-17T23:29:32.649 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:29:32.649 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:29:32.649 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-17T23:29:32.649 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:29:33.649 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 info mirror/image_37@snap 2024-09-17T23:29:33.696 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'image_37': 2024-09-17T23:29:33.696 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 128 MiB in 32 objects 2024-09-17T23:29:33.696 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:29:33.696 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 2 2024-09-17T23:29:33.696 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 12105230b1f9 2024-09-17T23:29:33.696 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.12105230b1f9 2024-09-17T23:29:33.696 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:29:33.696 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-17T23:29:33.697 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:29:33.697 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:29:33.697 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:22:43 2024 2024-09-17T23:29:33.697 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:22:43 2024 2024-09-17T23:29:33.697 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:22:43 2024 2024-09-17T23:29:33.697 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: protected: False 2024-09-17T23:29:33.697 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:29:33.697 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:29:33.697 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: ccc8bbca-2b19-4cff-8cb4-7770b06359dc 2024-09-17T23:29:33.697 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: false 2024-09-17T23:29:33.700 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:29:33.700 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ compare_image_snaps mirror image_37 snap 2024-09-17T23:29:33.700 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:29:33.700 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_37 2024-09-17T23:29:33.700 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:29:33.701 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local ret=0 2024-09-17T23:29:33.701 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local rmt_export=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_37.export 2024-09-17T23:29:33.701 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local loc_export=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_37.export 2024-09-17T23:29:33.701 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_37.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_37.export 2024-09-17T23:29:33.701 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror export image_37@snap /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_37.export 2024-09-17T23:29:33.876 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-17T23:29:33.883 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 -p mirror export image_37@snap /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_37.export 2024-09-17T23:29:34.109 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-17T23:29:34.114 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ cmp /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_37.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_37.export 2024-09-17T23:29:34.176 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_37.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_37.export 2024-09-17T23:29:34.209 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:29:34.209 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:29:34.209 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_38 2024-09-17T23:29:34.209 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_snap cluster2 mirror image_38 snap 2024-09-17T23:29:34.209 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:29:34.209 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:29:34.209 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_38 2024-09-17T23:29:34.209 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:29:34.209 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror snap create image_38@snap --debug-rbd=20 --debug-journaler=20 2024-09-17T23:29:36.025 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_started cluster1 mirror image_38 2024-09-17T23:29:36.025 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:29:36.025 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:29:36.026 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_38 2024-09-17T23:29:36.026 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_state cluster1 mirror image_38 started 2024-09-17T23:29:36.026 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:29:36.026 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:29:36.026 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_38 2024-09-17T23:29:36.026 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=started 2024-09-17T23:29:36.026 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:29:36.026 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:29:36.026 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:29:37.027 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_replay_state cluster1 mirror image_38 started 2024-09-17T23:29:37.027 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:29:37.027 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:29:37.027 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_38 2024-09-17T23:29:37.027 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=started 2024-09-17T23:29:37.027 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_result 2024-09-17T23:29:37.027 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=stopped 2024-09-17T23:29:37.028 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_38 2024-09-17T23:29:37.028 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_instance=cluster1 2024-09-17T23:29:37.028 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ grep -i state 2024-09-17T23:29:37.028 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:29:37.029 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:29:37.029 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance=cluster1 2024-09-17T23:29:37.029 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local loop_instance 2024-09-17T23:29:37.029 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:29:37.029 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sleep 0 2024-09-17T23:29:37.029 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-17T23:29:37.032 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ seq 0 3 2024-09-17T23:29:37.032 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:29:37.032 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_38 2024-09-17T23:29:37.032 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:0 2024-09-17T23:29:37.032 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:29:37.032 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:29:37.032 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-17T23:29:37.032 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:0 2024-09-17T23:29:37.033 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:29:37.033 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:29:37.033 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:29:37.033 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=0 2024-09-17T23:29:37.033 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-17T23:29:37.033 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:29:37.033 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:29:37.033 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=0 2024-09-17T23:29:37.033 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:29:37.033 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-17T23:29:37.033 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:0 2024-09-17T23:29:37.034 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:29:37.034 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:29:37.034 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-17T23:29:37.034 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:0 2024-09-17T23:29:37.034 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:29:37.034 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:29:37.034 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:29:37.034 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:29:37.034 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-17T23:29:37.034 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:29:37.034 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:29:37.035 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=0 2024-09-17T23:29:37.035 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=0 2024-09-17T23:29:37.035 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-17T23:29:37.053 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:29:37.054 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:29:37.054 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:29:37.054 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_38 2024-09-17T23:29:37.182 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ return 0 2024-09-17T23:29:37.183 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ status_result=' "state": "Replaying"' 2024-09-17T23:29:37.183 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ echo ' "state": "Replaying"' 2024-09-17T23:29:37.183 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep -i Replaying 2024-09-17T23:29:37.184 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: "state": "Replaying" 2024-09-17T23:29:37.185 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_state=started 2024-09-17T23:29:37.186 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test started = started 2024-09-17T23:29:37.186 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:29:37.186 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_38 2024-09-17T23:29:37.186 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:29:37.186 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:29:37.186 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:29:37.186 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_38 2024-09-17T23:29:37.186 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = journal ']' 2024-09-17T23:29:37.186 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = snapshot ']' 2024-09-17T23:29:37.186 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_38 2024-09-17T23:29:37.187 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:29:37.187 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:29:37.187 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:29:37.187 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_38 2024-09-17T23:29:37.187 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster2-mirror-image_38.status 2024-09-17T23:29:37.187 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster2-mirror-image_38.status 2024-09-17T23:29:37.187 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:29:37.188 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_38.status 2024-09-17T23:29:37.188 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster1-mirror-image_38.status 2024-09-17T23:29:37.188 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster1-mirror-image_38.status 2024-09-17T23:29:37.188 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:29:37.189 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_status_log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_38.status 2024-09-17T23:29:37.189 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ mirror_image_snapshot cluster2 mirror image_38 2024-09-17T23:29:37.189 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:29:37.189 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:29:37.189 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_38 2024-09-17T23:29:37.189 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_38 2024-09-17T23:29:38.039 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Snapshot ID: 160 2024-09-17T23:29:38.051 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_38 /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_38.status 2024-09-17T23:29:38.052 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:29:38.052 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:29:38.052 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_38 2024-09-17T23:29:38.052 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_38.status 2024-09-17T23:29:38.052 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_38 --format xml 2024-09-17T23:29:38.052 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:29:38.099 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-17T23:29:38.100 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snapshot_id=160 2024-09-17T23:29:38.101 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:29:38.101 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:29:38.101 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.2 2024-09-17T23:29:38.302 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_38 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_38.status 2024-09-17T23:29:38.302 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:29:38.302 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:29:38.302 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_38 2024-09-17T23:29:38.302 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_38.status 2024-09-17T23:29:38.302 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_38 --format xml 2024-09-17T23:29:38.303 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:29:38.352 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:29:38.352 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:29:38.353 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:29:38.353 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:29:38.353 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:29:38.354 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:29:38.354 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 160 = '' 2024-09-17T23:29:38.354 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:29:38.354 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.4 2024-09-17T23:29:38.755 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_38 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_38.status 2024-09-17T23:29:38.755 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:29:38.755 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:29:38.756 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_38 2024-09-17T23:29:38.756 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_38.status 2024-09-17T23:29:38.756 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_38 --format xml 2024-09-17T23:29:38.756 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:29:38.815 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:29:38.815 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:29:38.815 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:29:38.815 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:29:38.815 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:29:38.815 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:29:38.815 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 160 = '' 2024-09-17T23:29:38.815 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:29:38.815 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.8 2024-09-17T23:29:39.614 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_38 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_38.status 2024-09-17T23:29:39.614 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:29:39.614 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:29:39.614 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_38 2024-09-17T23:29:39.614 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_38.status 2024-09-17T23:29:39.614 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_38 --format xml 2024-09-17T23:29:39.614 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:29:39.661 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:29:39.663 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id=160 2024-09-17T23:29:39.663 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 160 = 160 2024-09-17T23:29:39.663 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:29:39.663 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snap_present cluster1 mirror image_38 snap 2024-09-17T23:29:39.663 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:29:39.663 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:29:39.663 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_38 2024-09-17T23:29:39.664 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:29:39.664 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:29:39.664 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-17T23:29:39.664 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:29:40.664 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 info mirror/image_38@snap 2024-09-17T23:29:40.714 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'image_38': 2024-09-17T23:29:40.714 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 128 MiB in 32 objects 2024-09-17T23:29:40.714 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:29:40.714 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 2 2024-09-17T23:29:40.714 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 121067f18e5 2024-09-17T23:29:40.714 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.121067f18e5 2024-09-17T23:29:40.714 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:29:40.714 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-17T23:29:40.714 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:29:40.715 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:29:40.715 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:22:53 2024 2024-09-17T23:29:40.715 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:22:53 2024 2024-09-17T23:29:40.715 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:22:53 2024 2024-09-17T23:29:40.715 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: protected: False 2024-09-17T23:29:40.715 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:29:40.715 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:29:40.715 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: 37d0dace-6324-49a7-9dcc-0827dcbac606 2024-09-17T23:29:40.715 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: false 2024-09-17T23:29:40.718 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:29:40.718 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ compare_image_snaps mirror image_38 snap 2024-09-17T23:29:40.718 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:29:40.718 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_38 2024-09-17T23:29:40.718 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:29:40.718 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local ret=0 2024-09-17T23:29:40.719 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local rmt_export=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_38.export 2024-09-17T23:29:40.719 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local loc_export=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_38.export 2024-09-17T23:29:40.719 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_38.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_38.export 2024-09-17T23:29:40.719 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror export image_38@snap /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_38.export 2024-09-17T23:29:40.882 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-17T23:29:40.889 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 -p mirror export image_38@snap /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_38.export 2024-09-17T23:29:41.300 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-17T23:29:41.307 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ cmp /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_38.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_38.export 2024-09-17T23:29:41.369 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_38.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_38.export 2024-09-17T23:29:41.401 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:29:41.401 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:29:41.401 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_39 2024-09-17T23:29:41.401 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_snap cluster2 mirror image_39 snap 2024-09-17T23:29:41.401 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:29:41.401 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:29:41.401 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_39 2024-09-17T23:29:41.401 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:29:41.401 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror snap create image_39@snap --debug-rbd=20 --debug-journaler=20 2024-09-17T23:29:41.847 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-09-17T23:29:41.848 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-09-17T23:29:41.848 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2024-09-17T23:29:41.849 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi038.stderr:2024-09-17T23:29:41.847+0000 7f844bde9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 43096) UID: 1000 2024-09-17T23:29:43.064 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_started cluster1 mirror image_39 2024-09-17T23:29:43.065 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:29:43.065 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:29:43.065 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_39 2024-09-17T23:29:43.065 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_state cluster1 mirror image_39 started 2024-09-17T23:29:43.065 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:29:43.065 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:29:43.065 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_39 2024-09-17T23:29:43.065 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=started 2024-09-17T23:29:43.065 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:29:43.065 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:29:43.065 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:29:44.066 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_replay_state cluster1 mirror image_39 started 2024-09-17T23:29:44.066 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:29:44.066 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:29:44.066 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_39 2024-09-17T23:29:44.066 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=started 2024-09-17T23:29:44.067 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_result 2024-09-17T23:29:44.067 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=stopped 2024-09-17T23:29:44.068 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_39 2024-09-17T23:29:44.068 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_instance=cluster1 2024-09-17T23:29:44.068 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:29:44.068 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ grep -i state 2024-09-17T23:29:44.068 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:29:44.068 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance=cluster1 2024-09-17T23:29:44.068 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local loop_instance 2024-09-17T23:29:44.068 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:29:44.068 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sleep 0 2024-09-17T23:29:44.068 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-17T23:29:44.071 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ seq 0 3 2024-09-17T23:29:44.071 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:29:44.071 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_39 2024-09-17T23:29:44.071 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:0 2024-09-17T23:29:44.071 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:29:44.071 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:29:44.071 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-17T23:29:44.071 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:0 2024-09-17T23:29:44.072 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:29:44.072 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:29:44.072 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:29:44.072 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=0 2024-09-17T23:29:44.072 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-17T23:29:44.072 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:29:44.072 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:29:44.072 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=0 2024-09-17T23:29:44.072 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:29:44.072 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-17T23:29:44.072 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:0 2024-09-17T23:29:44.072 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:29:44.072 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:29:44.072 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-17T23:29:44.073 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:0 2024-09-17T23:29:44.073 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:29:44.073 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:29:44.073 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:29:44.073 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:29:44.073 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-17T23:29:44.073 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:29:44.073 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:29:44.073 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=0 2024-09-17T23:29:44.073 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=0 2024-09-17T23:29:44.073 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-17T23:29:44.093 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:29:44.093 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:29:44.094 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:29:44.094 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_39 2024-09-17T23:29:44.167 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2024-09-17T23:29:44.171 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:29:44.171 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_39 2024-09-17T23:29:44.171 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:1 2024-09-17T23:29:44.171 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:29:44.171 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:29:44.171 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-17T23:29:44.171 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:1 2024-09-17T23:29:44.171 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:29:44.171 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:29:44.171 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:29:44.171 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=1 2024-09-17T23:29:44.171 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-17T23:29:44.171 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:29:44.172 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:29:44.172 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=1 2024-09-17T23:29:44.172 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=1 2024-09-17T23:29:44.172 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-17T23:29:44.172 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:1 2024-09-17T23:29:44.172 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:29:44.172 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:29:44.172 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-17T23:29:44.172 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:1 2024-09-17T23:29:44.172 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:29:44.172 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:29:44.172 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:29:44.172 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=1 2024-09-17T23:29:44.172 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-17T23:29:44.173 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:29:44.173 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:29:44.173 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=1 2024-09-17T23:29:44.173 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=1 2024-09-17T23:29:44.173 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-17T23:29:44.190 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-17T23:29:44.190 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-17T23:29:44.190 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-17T23:29:44.190 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_39 2024-09-17T23:29:44.268 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:Invalid command: unused arguments: ['mirror/image_39'] 2024-09-17T23:29:44.268 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd mirror status : get status for rbd mirror 2024-09-17T23:29:44.269 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:admin_socket: invalid command 2024-09-17T23:29:44.271 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:29:44.272 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_39 2024-09-17T23:29:44.272 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:2 2024-09-17T23:29:44.272 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:29:44.272 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:29:44.272 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-17T23:29:44.272 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:2 2024-09-17T23:29:44.272 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:29:44.272 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:29:44.272 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:29:44.272 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=2 2024-09-17T23:29:44.272 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-17T23:29:44.272 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:29:44.272 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:29:44.273 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=2 2024-09-17T23:29:44.273 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=2 2024-09-17T23:29:44.273 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-17T23:29:44.273 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:2 2024-09-17T23:29:44.273 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:29:44.273 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:29:44.273 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-17T23:29:44.273 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:2 2024-09-17T23:29:44.273 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:29:44.273 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:29:44.273 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:29:44.273 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=2 2024-09-17T23:29:44.273 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-17T23:29:44.274 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:29:44.274 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:29:44.274 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=2 2024-09-17T23:29:44.274 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=2 2024-09-17T23:29:44.274 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-17T23:29:44.290 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-17T23:29:44.290 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-17T23:29:44.291 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-17T23:29:44.291 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_39 2024-09-17T23:29:44.367 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:Invalid command: unused arguments: ['mirror/image_39'] 2024-09-17T23:29:44.367 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd mirror status : get status for rbd mirror 2024-09-17T23:29:44.367 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:admin_socket: invalid command 2024-09-17T23:29:44.370 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:29:44.371 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_39 2024-09-17T23:29:44.371 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:3 2024-09-17T23:29:44.371 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:29:44.371 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:29:44.371 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-17T23:29:44.371 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:3 2024-09-17T23:29:44.371 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:29:44.371 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:29:44.371 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:29:44.371 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=3 2024-09-17T23:29:44.371 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-17T23:29:44.372 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:29:44.372 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:29:44.372 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=3 2024-09-17T23:29:44.372 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=3 2024-09-17T23:29:44.372 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-17T23:29:44.372 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:3 2024-09-17T23:29:44.372 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:29:44.372 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:29:44.372 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-17T23:29:44.372 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:3 2024-09-17T23:29:44.372 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:29:44.372 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:29:44.373 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:29:44.373 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=3 2024-09-17T23:29:44.373 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-17T23:29:44.373 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:29:44.373 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:29:44.373 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=3 2024-09-17T23:29:44.373 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=3 2024-09-17T23:29:44.373 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-17T23:29:44.389 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-17T23:29:44.389 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-17T23:29:44.389 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-17T23:29:44.389 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_39 2024-09-17T23:29:44.464 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:Invalid command: unused arguments: ['mirror/image_39'] 2024-09-17T23:29:44.465 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd mirror status : get status for rbd mirror 2024-09-17T23:29:44.465 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:admin_socket: invalid command 2024-09-17T23:29:44.465 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:29:44.466 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sleep 1 2024-09-17T23:29:45.467 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-17T23:29:45.467 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ seq 0 3 2024-09-17T23:29:45.470 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:29:45.470 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_39 2024-09-17T23:29:45.470 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:0 2024-09-17T23:29:45.470 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:29:45.471 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:29:45.471 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-17T23:29:45.471 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:0 2024-09-17T23:29:45.471 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:29:45.471 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:29:45.471 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:29:45.471 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=0 2024-09-17T23:29:45.471 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-17T23:29:45.471 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:29:45.471 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:29:45.471 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=0 2024-09-17T23:29:45.471 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:29:45.471 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-17T23:29:45.471 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:0 2024-09-17T23:29:45.472 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:29:45.472 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:29:45.472 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-17T23:29:45.472 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:0 2024-09-17T23:29:45.472 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:29:45.472 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:29:45.472 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:29:45.472 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:29:45.472 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-17T23:29:45.472 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:29:45.472 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:29:45.472 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=0 2024-09-17T23:29:45.472 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=0 2024-09-17T23:29:45.473 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-17T23:29:45.492 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:29:45.493 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:29:45.493 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:29:45.493 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_39 2024-09-17T23:29:45.566 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2024-09-17T23:29:45.569 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:29:45.569 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_39 2024-09-17T23:29:45.569 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:1 2024-09-17T23:29:45.569 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:29:45.569 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:29:45.569 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-17T23:29:45.569 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:1 2024-09-17T23:29:45.569 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:29:45.570 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:29:45.570 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:29:45.570 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=1 2024-09-17T23:29:45.570 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-17T23:29:45.570 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:29:45.570 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:29:45.570 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=1 2024-09-17T23:29:45.570 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=1 2024-09-17T23:29:45.570 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-17T23:29:45.570 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:1 2024-09-17T23:29:45.570 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:29:45.570 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:29:45.570 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-17T23:29:45.571 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:1 2024-09-17T23:29:45.571 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:29:45.571 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:29:45.571 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:29:45.571 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=1 2024-09-17T23:29:45.571 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-17T23:29:45.571 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:29:45.571 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:29:45.571 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=1 2024-09-17T23:29:45.571 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=1 2024-09-17T23:29:45.571 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-17T23:29:45.589 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-17T23:29:45.589 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-17T23:29:45.589 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-17T23:29:45.589 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_39 2024-09-17T23:29:45.665 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:Invalid command: unused arguments: ['mirror/image_39'] 2024-09-17T23:29:45.666 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd mirror status : get status for rbd mirror 2024-09-17T23:29:45.666 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:admin_socket: invalid command 2024-09-17T23:29:45.668 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:29:45.668 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_39 2024-09-17T23:29:45.668 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:2 2024-09-17T23:29:45.669 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:29:45.669 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:29:45.669 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-17T23:29:45.669 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:2 2024-09-17T23:29:45.669 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:29:45.669 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:29:45.669 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:29:45.669 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=2 2024-09-17T23:29:45.669 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-17T23:29:45.669 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:29:45.669 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:29:45.669 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=2 2024-09-17T23:29:45.669 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=2 2024-09-17T23:29:45.670 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-17T23:29:45.670 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:2 2024-09-17T23:29:45.670 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:29:45.670 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:29:45.670 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-17T23:29:45.670 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:2 2024-09-17T23:29:45.670 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:29:45.670 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:29:45.670 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:29:45.670 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=2 2024-09-17T23:29:45.670 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-17T23:29:45.670 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:29:45.670 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:29:45.670 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=2 2024-09-17T23:29:45.671 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=2 2024-09-17T23:29:45.671 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-17T23:29:45.689 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-17T23:29:45.689 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-17T23:29:45.689 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-17T23:29:45.689 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_39 2024-09-17T23:29:45.764 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:Invalid command: unused arguments: ['mirror/image_39'] 2024-09-17T23:29:45.764 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd mirror status : get status for rbd mirror 2024-09-17T23:29:45.764 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:admin_socket: invalid command 2024-09-17T23:29:45.768 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:29:45.769 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_39 2024-09-17T23:29:45.769 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:3 2024-09-17T23:29:45.769 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:29:45.769 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:29:45.769 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-17T23:29:45.769 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:3 2024-09-17T23:29:45.769 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:29:45.769 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:29:45.769 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:29:45.769 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=3 2024-09-17T23:29:45.769 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-17T23:29:45.769 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:29:45.770 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:29:45.770 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=3 2024-09-17T23:29:45.770 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=3 2024-09-17T23:29:45.770 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-17T23:29:45.770 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:3 2024-09-17T23:29:45.770 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:29:45.770 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:29:45.770 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-17T23:29:45.770 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:3 2024-09-17T23:29:45.770 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:29:45.770 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:29:45.770 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:29:45.770 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=3 2024-09-17T23:29:45.770 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-17T23:29:45.770 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:29:45.771 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:29:45.771 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=3 2024-09-17T23:29:45.771 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=3 2024-09-17T23:29:45.771 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-17T23:29:45.786 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-17T23:29:45.786 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-17T23:29:45.786 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-17T23:29:45.786 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_39 2024-09-17T23:29:45.863 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:Invalid command: unused arguments: ['mirror/image_39'] 2024-09-17T23:29:45.863 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd mirror status : get status for rbd mirror 2024-09-17T23:29:45.863 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:admin_socket: invalid command 2024-09-17T23:29:45.864 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:29:45.864 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sleep 2 2024-09-17T23:29:47.866 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-17T23:29:47.866 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ seq 0 3 2024-09-17T23:29:47.869 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:29:47.869 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_39 2024-09-17T23:29:47.869 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:0 2024-09-17T23:29:47.869 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:29:47.869 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:29:47.869 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-17T23:29:47.869 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:0 2024-09-17T23:29:47.869 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:29:47.870 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:29:47.870 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:29:47.870 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=0 2024-09-17T23:29:47.870 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-17T23:29:47.870 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:29:47.870 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:29:47.870 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=0 2024-09-17T23:29:47.870 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:29:47.870 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-17T23:29:47.870 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:0 2024-09-17T23:29:47.870 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:29:47.870 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:29:47.870 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-17T23:29:47.871 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:0 2024-09-17T23:29:47.871 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:29:47.871 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:29:47.871 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:29:47.871 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:29:47.871 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-17T23:29:47.871 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:29:47.871 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:29:47.871 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=0 2024-09-17T23:29:47.871 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=0 2024-09-17T23:29:47.871 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-17T23:29:47.891 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:29:47.891 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:29:47.891 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:29:47.892 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_39 2024-09-17T23:29:47.965 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2024-09-17T23:29:47.968 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:29:47.969 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_39 2024-09-17T23:29:47.969 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:1 2024-09-17T23:29:47.969 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:29:47.969 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:29:47.969 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-17T23:29:47.969 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:1 2024-09-17T23:29:47.969 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:29:47.969 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:29:47.969 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:29:47.969 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=1 2024-09-17T23:29:47.969 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-17T23:29:47.969 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:29:47.970 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:29:47.970 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=1 2024-09-17T23:29:47.970 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=1 2024-09-17T23:29:47.970 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-17T23:29:47.970 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:1 2024-09-17T23:29:47.970 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:29:47.970 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:29:47.970 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-17T23:29:47.970 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:1 2024-09-17T23:29:47.970 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:29:47.970 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:29:47.970 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:29:47.971 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=1 2024-09-17T23:29:47.971 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-17T23:29:47.971 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:29:47.971 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:29:47.971 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=1 2024-09-17T23:29:47.971 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=1 2024-09-17T23:29:47.971 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-17T23:29:47.988 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-17T23:29:47.988 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-17T23:29:47.988 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-17T23:29:47.988 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_39 2024-09-17T23:29:48.065 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:Invalid command: unused arguments: ['mirror/image_39'] 2024-09-17T23:29:48.065 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd mirror status : get status for rbd mirror 2024-09-17T23:29:48.065 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:admin_socket: invalid command 2024-09-17T23:29:48.068 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:29:48.068 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_39 2024-09-17T23:29:48.068 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:2 2024-09-17T23:29:48.068 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:29:48.068 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:29:48.068 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-17T23:29:48.068 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:2 2024-09-17T23:29:48.068 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:29:48.069 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:29:48.069 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:29:48.069 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=2 2024-09-17T23:29:48.069 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-17T23:29:48.069 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:29:48.069 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:29:48.069 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=2 2024-09-17T23:29:48.069 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=2 2024-09-17T23:29:48.069 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-17T23:29:48.069 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:2 2024-09-17T23:29:48.069 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:29:48.069 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:29:48.069 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-17T23:29:48.069 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:2 2024-09-17T23:29:48.069 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:29:48.070 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:29:48.070 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:29:48.070 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=2 2024-09-17T23:29:48.070 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-17T23:29:48.070 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:29:48.070 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:29:48.070 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=2 2024-09-17T23:29:48.070 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=2 2024-09-17T23:29:48.070 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-17T23:29:48.087 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-17T23:29:48.087 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-17T23:29:48.087 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-17T23:29:48.087 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_39 2024-09-17T23:29:48.206 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ return 0 2024-09-17T23:29:48.207 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ status_result=' "state": "Replaying"' 2024-09-17T23:29:48.207 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ echo ' "state": "Replaying"' 2024-09-17T23:29:48.207 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep -i Replaying 2024-09-17T23:29:48.208 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: "state": "Replaying" 2024-09-17T23:29:48.209 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_state=started 2024-09-17T23:29:48.209 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test started = started 2024-09-17T23:29:48.209 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:29:48.209 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_39 2024-09-17T23:29:48.209 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:29:48.210 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:29:48.210 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:29:48.210 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_39 2024-09-17T23:29:48.210 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = journal ']' 2024-09-17T23:29:48.210 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = snapshot ']' 2024-09-17T23:29:48.210 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_39 2024-09-17T23:29:48.210 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:29:48.210 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:29:48.210 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:29:48.210 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_39 2024-09-17T23:29:48.210 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster2-mirror-image_39.status 2024-09-17T23:29:48.210 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster2-mirror-image_39.status 2024-09-17T23:29:48.210 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:29:48.211 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_39.status 2024-09-17T23:29:48.211 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster1-mirror-image_39.status 2024-09-17T23:29:48.211 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster1-mirror-image_39.status 2024-09-17T23:29:48.211 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:29:48.212 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_status_log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_39.status 2024-09-17T23:29:48.212 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ mirror_image_snapshot cluster2 mirror image_39 2024-09-17T23:29:48.212 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:29:48.212 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:29:48.212 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_39 2024-09-17T23:29:48.212 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_39 2024-09-17T23:29:48.849 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-09-17T23:29:48.849 INFO:teuthology.orchestra.run:waiting for 600 2024-09-17T23:29:48.849 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-09-17T23:29:48.849 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-09-17T23:29:48.850 DEBUG:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-09-17T23:29:48.852 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-09-17T23:29:48.853 INFO:tasks.rbd_mirror_thrash:waiting for 95 secs before thrashing 2024-09-17T23:29:49.069 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Snapshot ID: 162 2024-09-17T23:29:49.079 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_39 /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_39.status 2024-09-17T23:29:49.080 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:29:49.080 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:29:49.080 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_39 2024-09-17T23:29:49.080 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_39.status 2024-09-17T23:29:49.080 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_39 --format xml 2024-09-17T23:29:49.080 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:29:49.124 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-17T23:29:49.126 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snapshot_id=162 2024-09-17T23:29:49.126 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:29:49.126 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:29:49.126 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.2 2024-09-17T23:29:49.328 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_39 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_39.status 2024-09-17T23:29:49.328 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:29:49.328 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:29:49.328 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_39 2024-09-17T23:29:49.329 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_39.status 2024-09-17T23:29:49.329 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_39 --format xml 2024-09-17T23:29:49.329 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:29:49.377 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:29:49.377 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:29:49.379 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:29:49.379 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:29:49.379 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:29:49.379 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:29:49.379 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 162 = '' 2024-09-17T23:29:49.379 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:29:49.379 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.4 2024-09-17T23:29:49.780 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_39 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_39.status 2024-09-17T23:29:49.780 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:29:49.780 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:29:49.780 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_39 2024-09-17T23:29:49.780 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_39.status 2024-09-17T23:29:49.780 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_39 --format xml 2024-09-17T23:29:49.780 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:29:49.830 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:29:49.830 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:29:49.832 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:29:49.832 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:29:49.832 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:29:49.832 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:29:49.832 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 162 = '' 2024-09-17T23:29:49.833 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:29:49.833 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.8 2024-09-17T23:29:50.633 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_39 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_39.status 2024-09-17T23:29:50.634 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:29:50.634 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:29:50.634 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_39 2024-09-17T23:29:50.634 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_39.status 2024-09-17T23:29:50.634 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_39 --format xml 2024-09-17T23:29:50.634 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:29:50.681 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:29:50.683 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id=162 2024-09-17T23:29:50.683 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 162 = 162 2024-09-17T23:29:50.683 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:29:50.684 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snap_present cluster1 mirror image_39 snap 2024-09-17T23:29:50.684 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:29:50.684 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:29:50.684 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_39 2024-09-17T23:29:50.684 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:29:50.684 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:29:50.684 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-17T23:29:50.684 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:29:51.684 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 info mirror/image_39@snap 2024-09-17T23:29:51.733 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'image_39': 2024-09-17T23:29:51.733 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 128 MiB in 32 objects 2024-09-17T23:29:51.733 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:29:51.733 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 2 2024-09-17T23:29:51.733 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 1210416083f1 2024-09-17T23:29:51.733 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.1210416083f1 2024-09-17T23:29:51.733 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:29:51.733 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-17T23:29:51.733 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:29:51.734 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:29:51.734 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:23:01 2024 2024-09-17T23:29:51.734 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:23:01 2024 2024-09-17T23:29:51.734 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:23:01 2024 2024-09-17T23:29:51.734 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: protected: False 2024-09-17T23:29:51.734 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:29:51.734 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:29:51.734 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: a0492a7c-0945-424f-9742-2a83199e6bda 2024-09-17T23:29:51.734 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: false 2024-09-17T23:29:51.737 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:29:51.737 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ compare_image_snaps mirror image_39 snap 2024-09-17T23:29:51.737 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:29:51.737 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_39 2024-09-17T23:29:51.737 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:29:51.737 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local ret=0 2024-09-17T23:29:51.737 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local rmt_export=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_39.export 2024-09-17T23:29:51.737 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local loc_export=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_39.export 2024-09-17T23:29:51.737 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_39.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_39.export 2024-09-17T23:29:51.738 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror export image_39@snap /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_39.export 2024-09-17T23:29:51.901 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-17T23:29:51.908 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 -p mirror export image_39@snap /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_39.export 2024-09-17T23:29:52.153 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-17T23:29:52.160 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ cmp /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_39.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_39.export 2024-09-17T23:29:52.223 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_39.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_39.export 2024-09-17T23:29:52.254 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:29:52.254 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:29:52.255 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_40 2024-09-17T23:29:52.255 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_snap cluster2 mirror image_40 snap 2024-09-17T23:29:52.255 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:29:52.255 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:29:52.255 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_40 2024-09-17T23:29:52.255 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:29:52.255 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror snap create image_40@snap --debug-rbd=20 --debug-journaler=20 2024-09-17T23:29:54.098 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_started cluster1 mirror image_40 2024-09-17T23:29:54.099 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:29:54.099 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:29:54.099 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_40 2024-09-17T23:29:54.099 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_state cluster1 mirror image_40 started 2024-09-17T23:29:54.099 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:29:54.099 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:29:54.099 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_40 2024-09-17T23:29:54.099 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=started 2024-09-17T23:29:54.099 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:29:54.099 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:29:54.099 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:29:55.102 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_replay_state cluster1 mirror image_40 started 2024-09-17T23:29:55.103 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:29:55.103 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:29:55.103 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_40 2024-09-17T23:29:55.103 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=started 2024-09-17T23:29:55.104 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_result 2024-09-17T23:29:55.104 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=stopped 2024-09-17T23:29:55.104 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_40 2024-09-17T23:29:55.104 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_instance=cluster1 2024-09-17T23:29:55.105 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:29:55.105 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ grep -i state 2024-09-17T23:29:55.105 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:29:55.105 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance=cluster1 2024-09-17T23:29:55.105 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local loop_instance 2024-09-17T23:29:55.105 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:29:55.106 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sleep 0 2024-09-17T23:29:55.106 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-17T23:29:55.106 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ seq 0 3 2024-09-17T23:29:55.109 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:29:55.109 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_40 2024-09-17T23:29:55.110 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:0 2024-09-17T23:29:55.110 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:29:55.110 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:29:55.110 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-17T23:29:55.110 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:0 2024-09-17T23:29:55.110 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:29:55.110 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:29:55.110 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:29:55.110 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=0 2024-09-17T23:29:55.111 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-17T23:29:55.111 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:29:55.111 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:29:55.111 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=0 2024-09-17T23:29:55.111 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:29:55.111 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-17T23:29:55.111 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:0 2024-09-17T23:29:55.111 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:29:55.111 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:29:55.111 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-17T23:29:55.111 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:0 2024-09-17T23:29:55.112 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:29:55.112 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:29:55.112 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:29:55.112 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:29:55.112 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-17T23:29:55.112 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:29:55.112 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:29:55.112 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=0 2024-09-17T23:29:55.112 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=0 2024-09-17T23:29:55.112 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-17T23:29:55.128 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:29:55.128 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:29:55.128 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:29:55.128 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_40 2024-09-17T23:29:55.211 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:Invalid command: unused arguments: ['mirror/image_40'] 2024-09-17T23:29:55.211 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd mirror status : get status for rbd mirror 2024-09-17T23:29:55.211 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:admin_socket: invalid command 2024-09-17T23:29:55.214 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:29:55.214 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_40 2024-09-17T23:29:55.214 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:1 2024-09-17T23:29:55.214 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:29:55.214 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:29:55.214 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-17T23:29:55.214 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:1 2024-09-17T23:29:55.214 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:29:55.214 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:29:55.214 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:29:55.214 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=1 2024-09-17T23:29:55.215 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-17T23:29:55.215 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:29:55.215 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:29:55.215 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=1 2024-09-17T23:29:55.215 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=1 2024-09-17T23:29:55.215 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-17T23:29:55.216 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:1 2024-09-17T23:29:55.216 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:29:55.216 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:29:55.216 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-17T23:29:55.216 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:1 2024-09-17T23:29:55.216 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:29:55.216 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:29:55.216 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:29:55.216 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=1 2024-09-17T23:29:55.216 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-17T23:29:55.216 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:29:55.216 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:29:55.217 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=1 2024-09-17T23:29:55.217 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=1 2024-09-17T23:29:55.217 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-17T23:29:55.233 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-17T23:29:55.234 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-17T23:29:55.234 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-17T23:29:55.234 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_40 2024-09-17T23:29:55.310 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:Invalid command: unused arguments: ['mirror/image_40'] 2024-09-17T23:29:55.310 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd mirror status : get status for rbd mirror 2024-09-17T23:29:55.311 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:admin_socket: invalid command 2024-09-17T23:29:55.313 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:29:55.314 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_40 2024-09-17T23:29:55.314 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:2 2024-09-17T23:29:55.314 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:29:55.314 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:29:55.314 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-17T23:29:55.314 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:2 2024-09-17T23:29:55.314 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:29:55.314 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:29:55.315 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:29:55.315 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=2 2024-09-17T23:29:55.316 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-17T23:29:55.316 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:29:55.316 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:29:55.316 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=2 2024-09-17T23:29:55.316 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=2 2024-09-17T23:29:55.316 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-17T23:29:55.316 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:2 2024-09-17T23:29:55.316 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:29:55.316 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:29:55.316 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-17T23:29:55.316 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:2 2024-09-17T23:29:55.317 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:29:55.317 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:29:55.317 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:29:55.317 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=2 2024-09-17T23:29:55.317 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-17T23:29:55.317 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:29:55.317 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:29:55.317 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=2 2024-09-17T23:29:55.317 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=2 2024-09-17T23:29:55.317 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-17T23:29:55.332 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-17T23:29:55.332 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-17T23:29:55.332 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-17T23:29:55.332 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_40 2024-09-17T23:29:55.452 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ return 0 2024-09-17T23:29:55.453 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ status_result=' "state": "Replaying"' 2024-09-17T23:29:55.453 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ echo ' "state": "Replaying"' 2024-09-17T23:29:55.453 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep -i Replaying 2024-09-17T23:29:55.454 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: "state": "Replaying" 2024-09-17T23:29:55.455 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_state=started 2024-09-17T23:29:55.455 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test started = started 2024-09-17T23:29:55.455 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:29:55.455 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_40 2024-09-17T23:29:55.456 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:29:55.456 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:29:55.456 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:29:55.456 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_40 2024-09-17T23:29:55.456 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = journal ']' 2024-09-17T23:29:55.456 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = snapshot ']' 2024-09-17T23:29:55.456 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_40 2024-09-17T23:29:55.456 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:29:55.456 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:29:55.456 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:29:55.456 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_40 2024-09-17T23:29:55.456 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster2-mirror-image_40.status 2024-09-17T23:29:55.457 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster2-mirror-image_40.status 2024-09-17T23:29:55.457 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:29:55.457 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_40.status 2024-09-17T23:29:55.457 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster1-mirror-image_40.status 2024-09-17T23:29:55.458 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster1-mirror-image_40.status 2024-09-17T23:29:55.458 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:29:55.458 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_status_log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_40.status 2024-09-17T23:29:55.458 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ mirror_image_snapshot cluster2 mirror image_40 2024-09-17T23:29:55.458 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:29:55.458 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:29:55.458 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_40 2024-09-17T23:29:55.458 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_40 2024-09-17T23:29:56.121 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Snapshot ID: 164 2024-09-17T23:29:56.134 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_40 /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_40.status 2024-09-17T23:29:56.134 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:29:56.134 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:29:56.134 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_40 2024-09-17T23:29:56.134 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_40.status 2024-09-17T23:29:56.135 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_40 --format xml 2024-09-17T23:29:56.135 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:29:56.183 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-17T23:29:56.185 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snapshot_id=164 2024-09-17T23:29:56.185 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:29:56.185 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:29:56.185 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.2 2024-09-17T23:29:56.387 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_40 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_40.status 2024-09-17T23:29:56.387 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:29:56.387 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:29:56.387 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_40 2024-09-17T23:29:56.387 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_40.status 2024-09-17T23:29:56.387 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_40 --format xml 2024-09-17T23:29:56.387 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:29:56.439 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:29:56.439 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:29:56.441 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:29:56.441 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:29:56.441 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:29:56.441 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:29:56.442 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 164 = '' 2024-09-17T23:29:56.442 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:29:56.442 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.4 2024-09-17T23:29:56.843 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_40 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_40.status 2024-09-17T23:29:56.843 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:29:56.843 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:29:56.843 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_40 2024-09-17T23:29:56.843 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_40.status 2024-09-17T23:29:56.843 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_40 --format xml 2024-09-17T23:29:56.843 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:29:56.898 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:29:56.898 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:29:56.899 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:29:56.899 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:29:56.900 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:29:56.900 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:29:56.900 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 164 = '' 2024-09-17T23:29:56.900 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:29:56.900 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.8 2024-09-17T23:29:57.702 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_40 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_40.status 2024-09-17T23:29:57.702 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:29:57.702 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:29:57.702 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_40 2024-09-17T23:29:57.702 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_40.status 2024-09-17T23:29:57.702 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_40 --format xml 2024-09-17T23:29:57.702 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:29:57.749 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:29:57.751 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id=164 2024-09-17T23:29:57.751 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 164 = 164 2024-09-17T23:29:57.751 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:29:57.751 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snap_present cluster1 mirror image_40 snap 2024-09-17T23:29:57.751 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:29:57.751 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:29:57.751 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_40 2024-09-17T23:29:57.751 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:29:57.752 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:29:57.752 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-17T23:29:57.752 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:29:58.752 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 info mirror/image_40@snap 2024-09-17T23:29:58.802 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'image_40': 2024-09-17T23:29:58.802 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 128 MiB in 32 objects 2024-09-17T23:29:58.802 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:29:58.802 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 2 2024-09-17T23:29:58.802 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 1210173e6374 2024-09-17T23:29:58.802 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.1210173e6374 2024-09-17T23:29:58.802 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:29:58.802 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-17T23:29:58.802 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:29:58.803 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:29:58.803 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:23:14 2024 2024-09-17T23:29:58.803 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:23:14 2024 2024-09-17T23:29:58.803 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:23:14 2024 2024-09-17T23:29:58.803 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: protected: False 2024-09-17T23:29:58.803 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:29:58.803 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:29:58.803 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: 2fcc3df0-03d9-483f-9f1f-787ec2efd74d 2024-09-17T23:29:58.803 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: false 2024-09-17T23:29:58.806 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:29:58.806 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ compare_image_snaps mirror image_40 snap 2024-09-17T23:29:58.806 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:29:58.806 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_40 2024-09-17T23:29:58.806 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:29:58.806 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local ret=0 2024-09-17T23:29:58.806 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local rmt_export=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_40.export 2024-09-17T23:29:58.806 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local loc_export=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_40.export 2024-09-17T23:29:58.806 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_40.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_40.export 2024-09-17T23:29:58.807 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror export image_40@snap /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_40.export 2024-09-17T23:29:58.975 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-17T23:29:58.982 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 -p mirror export image_40@snap /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_40.export 2024-09-17T23:29:59.207 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-17T23:29:59.214 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ cmp /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_40.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_40.export 2024-09-17T23:29:59.276 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_40.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_40.export 2024-09-17T23:29:59.307 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:29:59.308 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:29:59.308 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_41 2024-09-17T23:29:59.308 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_snap cluster2 mirror image_41 snap 2024-09-17T23:29:59.308 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:29:59.308 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:29:59.308 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_41 2024-09-17T23:29:59.308 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:29:59.308 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror snap create image_41@snap --debug-rbd=20 --debug-journaler=20 2024-09-17T23:30:01.143 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_started cluster1 mirror image_41 2024-09-17T23:30:01.143 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:30:01.143 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:30:01.143 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_41 2024-09-17T23:30:01.143 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_state cluster1 mirror image_41 started 2024-09-17T23:30:01.143 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:30:01.144 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:30:01.144 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_41 2024-09-17T23:30:01.144 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=started 2024-09-17T23:30:01.144 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:30:01.144 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:30:01.144 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:30:02.144 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_replay_state cluster1 mirror image_41 started 2024-09-17T23:30:02.144 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:30:02.145 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:30:02.145 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_41 2024-09-17T23:30:02.145 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=started 2024-09-17T23:30:02.145 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_result 2024-09-17T23:30:02.145 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=stopped 2024-09-17T23:30:02.146 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_41 2024-09-17T23:30:02.146 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_instance=cluster1 2024-09-17T23:30:02.146 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:30:02.146 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ grep -i state 2024-09-17T23:30:02.146 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:30:02.146 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance=cluster1 2024-09-17T23:30:02.146 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local loop_instance 2024-09-17T23:30:02.146 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:30:02.147 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sleep 0 2024-09-17T23:30:02.147 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-17T23:30:02.147 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ seq 0 3 2024-09-17T23:30:02.149 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:30:02.150 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_41 2024-09-17T23:30:02.150 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:0 2024-09-17T23:30:02.150 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:30:02.150 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:30:02.150 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-17T23:30:02.150 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:0 2024-09-17T23:30:02.150 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:30:02.150 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:30:02.150 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:30:02.150 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=0 2024-09-17T23:30:02.150 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-17T23:30:02.150 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:30:02.151 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:30:02.151 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=0 2024-09-17T23:30:02.151 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:30:02.151 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-17T23:30:02.151 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:0 2024-09-17T23:30:02.151 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:30:02.151 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:30:02.151 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-17T23:30:02.151 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:0 2024-09-17T23:30:02.151 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:30:02.151 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:30:02.151 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:30:02.151 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:30:02.152 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-17T23:30:02.152 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:30:02.152 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:30:02.152 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=0 2024-09-17T23:30:02.152 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=0 2024-09-17T23:30:02.152 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-17T23:30:02.172 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:30:02.172 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:30:02.172 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:30:02.172 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_41 2024-09-17T23:30:02.255 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:Invalid command: unused arguments: ['mirror/image_41'] 2024-09-17T23:30:02.255 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd mirror status : get status for rbd mirror 2024-09-17T23:30:02.255 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:admin_socket: invalid command 2024-09-17T23:30:02.258 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:30:02.258 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_41 2024-09-17T23:30:02.258 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:1 2024-09-17T23:30:02.259 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:30:02.259 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:30:02.259 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-17T23:30:02.259 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:1 2024-09-17T23:30:02.259 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:30:02.259 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:30:02.259 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:30:02.259 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=1 2024-09-17T23:30:02.259 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-17T23:30:02.259 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:30:02.259 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:30:02.259 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=1 2024-09-17T23:30:02.259 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=1 2024-09-17T23:30:02.260 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-17T23:30:02.260 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:1 2024-09-17T23:30:02.260 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:30:02.260 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:30:02.260 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-17T23:30:02.260 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:1 2024-09-17T23:30:02.260 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:30:02.260 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:30:02.260 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:30:02.260 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=1 2024-09-17T23:30:02.260 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-17T23:30:02.260 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:30:02.261 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:30:02.261 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=1 2024-09-17T23:30:02.261 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=1 2024-09-17T23:30:02.261 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-17T23:30:02.279 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-17T23:30:02.279 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-17T23:30:02.279 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-17T23:30:02.279 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_41 2024-09-17T23:30:02.354 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:Invalid command: unused arguments: ['mirror/image_41'] 2024-09-17T23:30:02.354 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd mirror status : get status for rbd mirror 2024-09-17T23:30:02.354 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:admin_socket: invalid command 2024-09-17T23:30:02.357 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:30:02.357 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_41 2024-09-17T23:30:02.357 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:2 2024-09-17T23:30:02.357 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:30:02.357 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:30:02.357 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-17T23:30:02.358 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:2 2024-09-17T23:30:02.358 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:30:02.358 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:30:02.358 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:30:02.358 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=2 2024-09-17T23:30:02.358 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-17T23:30:02.358 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:30:02.358 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:30:02.358 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=2 2024-09-17T23:30:02.358 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=2 2024-09-17T23:30:02.358 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-17T23:30:02.358 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:2 2024-09-17T23:30:02.358 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:30:02.359 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:30:02.359 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-17T23:30:02.359 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:2 2024-09-17T23:30:02.359 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:30:02.359 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:30:02.359 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:30:02.359 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=2 2024-09-17T23:30:02.359 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-17T23:30:02.359 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:30:02.359 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:30:02.360 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=2 2024-09-17T23:30:02.360 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=2 2024-09-17T23:30:02.360 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-17T23:30:02.375 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-17T23:30:02.376 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-17T23:30:02.376 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-17T23:30:02.376 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_41 2024-09-17T23:30:02.496 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ return 0 2024-09-17T23:30:02.496 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ status_result=' "state": "Replaying"' 2024-09-17T23:30:02.496 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ echo ' "state": "Replaying"' 2024-09-17T23:30:02.497 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep -i Replaying 2024-09-17T23:30:02.497 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: "state": "Replaying" 2024-09-17T23:30:02.499 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_state=started 2024-09-17T23:30:02.500 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test started = started 2024-09-17T23:30:02.500 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:30:02.500 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_41 2024-09-17T23:30:02.500 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:30:02.500 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:30:02.500 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:30:02.500 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_41 2024-09-17T23:30:02.500 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = journal ']' 2024-09-17T23:30:02.500 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = snapshot ']' 2024-09-17T23:30:02.500 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_41 2024-09-17T23:30:02.500 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:30:02.501 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:30:02.501 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:30:02.501 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_41 2024-09-17T23:30:02.501 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster2-mirror-image_41.status 2024-09-17T23:30:02.501 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster2-mirror-image_41.status 2024-09-17T23:30:02.501 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:30:02.501 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_41.status 2024-09-17T23:30:02.501 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster1-mirror-image_41.status 2024-09-17T23:30:02.501 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster1-mirror-image_41.status 2024-09-17T23:30:02.501 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:30:02.502 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_status_log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_41.status 2024-09-17T23:30:02.502 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ mirror_image_snapshot cluster2 mirror image_41 2024-09-17T23:30:02.502 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:30:02.502 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:30:02.503 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_41 2024-09-17T23:30:02.503 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_41 2024-09-17T23:30:03.143 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Snapshot ID: 166 2024-09-17T23:30:03.156 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_41 /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_41.status 2024-09-17T23:30:03.156 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:30:03.156 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:30:03.156 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_41 2024-09-17T23:30:03.157 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_41.status 2024-09-17T23:30:03.157 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_41 --format xml 2024-09-17T23:30:03.157 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:30:03.204 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-17T23:30:03.206 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snapshot_id=166 2024-09-17T23:30:03.206 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:30:03.206 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:30:03.206 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.2 2024-09-17T23:30:03.407 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_41 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_41.status 2024-09-17T23:30:03.407 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:30:03.408 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:30:03.408 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_41 2024-09-17T23:30:03.408 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_41.status 2024-09-17T23:30:03.408 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_41 --format xml 2024-09-17T23:30:03.408 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:30:03.457 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:30:03.457 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:30:03.459 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:30:03.459 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:30:03.459 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:30:03.459 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:30:03.459 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 166 = '' 2024-09-17T23:30:03.459 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:30:03.460 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.4 2024-09-17T23:30:03.861 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_41 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_41.status 2024-09-17T23:30:03.861 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:30:03.861 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:30:03.861 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_41 2024-09-17T23:30:03.861 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_41.status 2024-09-17T23:30:03.861 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_41 --format xml 2024-09-17T23:30:03.861 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:30:03.913 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:30:03.913 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:30:03.915 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:30:03.915 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:30:03.915 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:30:03.915 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:30:03.915 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 166 = '' 2024-09-17T23:30:03.916 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:30:03.916 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.8 2024-09-17T23:30:04.717 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_41 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_41.status 2024-09-17T23:30:04.717 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:30:04.717 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:30:04.717 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_41 2024-09-17T23:30:04.717 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_41.status 2024-09-17T23:30:04.717 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_41 --format xml 2024-09-17T23:30:04.717 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:30:04.766 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:30:04.768 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id=166 2024-09-17T23:30:04.768 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 166 = 166 2024-09-17T23:30:04.768 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:30:04.768 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snap_present cluster1 mirror image_41 snap 2024-09-17T23:30:04.768 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:30:04.768 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:30:04.768 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_41 2024-09-17T23:30:04.768 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:30:04.768 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:30:04.768 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-17T23:30:04.768 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:30:05.769 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 info mirror/image_41@snap 2024-09-17T23:30:05.817 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'image_41': 2024-09-17T23:30:05.817 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 128 MiB in 32 objects 2024-09-17T23:30:05.817 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:30:05.817 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 2 2024-09-17T23:30:05.817 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 12103b90882c 2024-09-17T23:30:05.817 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.12103b90882c 2024-09-17T23:30:05.817 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:30:05.817 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-17T23:30:05.817 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:30:05.817 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:30:05.817 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:23:28 2024 2024-09-17T23:30:05.818 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:23:28 2024 2024-09-17T23:30:05.818 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:23:28 2024 2024-09-17T23:30:05.818 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: protected: False 2024-09-17T23:30:05.818 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:30:05.818 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:30:05.818 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: 7a67405f-a922-4db6-a34d-aef043e8bddf 2024-09-17T23:30:05.818 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: false 2024-09-17T23:30:05.820 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:30:05.820 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ compare_image_snaps mirror image_41 snap 2024-09-17T23:30:05.820 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:30:05.820 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_41 2024-09-17T23:30:05.821 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:30:05.821 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local ret=0 2024-09-17T23:30:05.821 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local rmt_export=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_41.export 2024-09-17T23:30:05.821 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local loc_export=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_41.export 2024-09-17T23:30:05.821 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_41.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_41.export 2024-09-17T23:30:05.821 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror export image_41@snap /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_41.export 2024-09-17T23:30:05.995 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-17T23:30:06.003 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 -p mirror export image_41@snap /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_41.export 2024-09-17T23:30:06.239 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-17T23:30:06.246 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ cmp /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_41.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_41.export 2024-09-17T23:30:06.308 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_41.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_41.export 2024-09-17T23:30:06.339 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:30:06.340 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:30:06.340 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_42 2024-09-17T23:30:06.340 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_snap cluster2 mirror image_42 snap 2024-09-17T23:30:06.340 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:30:06.340 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:30:06.340 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_42 2024-09-17T23:30:06.340 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:30:06.340 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror snap create image_42@snap --debug-rbd=20 --debug-journaler=20 2024-09-17T23:30:08.175 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_started cluster1 mirror image_42 2024-09-17T23:30:08.175 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:30:08.175 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:30:08.176 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_42 2024-09-17T23:30:08.176 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_state cluster1 mirror image_42 started 2024-09-17T23:30:08.176 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:30:08.176 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:30:08.176 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_42 2024-09-17T23:30:08.176 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=started 2024-09-17T23:30:08.176 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:30:08.176 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:30:08.176 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:30:09.176 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_replay_state cluster1 mirror image_42 started 2024-09-17T23:30:09.177 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:30:09.177 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:30:09.177 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_42 2024-09-17T23:30:09.177 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=started 2024-09-17T23:30:09.177 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_result 2024-09-17T23:30:09.177 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=stopped 2024-09-17T23:30:09.178 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_42 2024-09-17T23:30:09.178 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_instance=cluster1 2024-09-17T23:30:09.178 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:30:09.178 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:30:09.178 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ grep -i state 2024-09-17T23:30:09.178 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance=cluster1 2024-09-17T23:30:09.178 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local loop_instance 2024-09-17T23:30:09.178 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:30:09.179 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sleep 0 2024-09-17T23:30:09.179 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-17T23:30:09.180 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ seq 0 3 2024-09-17T23:30:09.180 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:30:09.180 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_42 2024-09-17T23:30:09.180 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:0 2024-09-17T23:30:09.181 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:30:09.181 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:30:09.181 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-17T23:30:09.181 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:0 2024-09-17T23:30:09.181 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:30:09.181 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:30:09.181 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:30:09.181 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=0 2024-09-17T23:30:09.181 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-17T23:30:09.181 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:30:09.181 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:30:09.181 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=0 2024-09-17T23:30:09.182 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:30:09.182 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-17T23:30:09.183 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:0 2024-09-17T23:30:09.183 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:30:09.183 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:30:09.183 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-17T23:30:09.183 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:0 2024-09-17T23:30:09.183 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:30:09.183 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:30:09.183 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:30:09.183 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:30:09.183 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-17T23:30:09.183 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:30:09.183 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:30:09.184 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=0 2024-09-17T23:30:09.184 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=0 2024-09-17T23:30:09.184 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-17T23:30:09.204 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:30:09.204 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:30:09.204 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:30:09.204 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_42 2024-09-17T23:30:09.286 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:Invalid command: unused arguments: ['mirror/image_42'] 2024-09-17T23:30:09.286 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd mirror status : get status for rbd mirror 2024-09-17T23:30:09.286 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:admin_socket: invalid command 2024-09-17T23:30:09.289 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:30:09.289 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_42 2024-09-17T23:30:09.289 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:1 2024-09-17T23:30:09.289 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:30:09.289 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:30:09.289 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-17T23:30:09.289 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:1 2024-09-17T23:30:09.290 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:30:09.290 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:30:09.290 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:30:09.290 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=1 2024-09-17T23:30:09.290 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-17T23:30:09.290 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:30:09.290 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:30:09.290 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=1 2024-09-17T23:30:09.290 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=1 2024-09-17T23:30:09.290 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-17T23:30:09.290 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:1 2024-09-17T23:30:09.290 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:30:09.290 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:30:09.290 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-17T23:30:09.291 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:1 2024-09-17T23:30:09.291 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:30:09.291 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:30:09.291 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:30:09.291 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=1 2024-09-17T23:30:09.291 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-17T23:30:09.291 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:30:09.291 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:30:09.291 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=1 2024-09-17T23:30:09.291 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=1 2024-09-17T23:30:09.291 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-17T23:30:09.308 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-17T23:30:09.308 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-17T23:30:09.309 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-17T23:30:09.309 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_42 2024-09-17T23:30:09.386 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:Invalid command: unused arguments: ['mirror/image_42'] 2024-09-17T23:30:09.386 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd mirror status : get status for rbd mirror 2024-09-17T23:30:09.386 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:admin_socket: invalid command 2024-09-17T23:30:09.389 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:30:09.389 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_42 2024-09-17T23:30:09.389 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:2 2024-09-17T23:30:09.389 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:30:09.390 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:30:09.390 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-17T23:30:09.390 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:2 2024-09-17T23:30:09.390 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:30:09.390 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:30:09.390 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:30:09.390 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=2 2024-09-17T23:30:09.390 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-17T23:30:09.390 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:30:09.390 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:30:09.390 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=2 2024-09-17T23:30:09.390 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=2 2024-09-17T23:30:09.390 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-17T23:30:09.390 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:2 2024-09-17T23:30:09.390 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:30:09.391 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:30:09.391 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-17T23:30:09.391 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:2 2024-09-17T23:30:09.391 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:30:09.391 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:30:09.391 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:30:09.391 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=2 2024-09-17T23:30:09.391 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-17T23:30:09.391 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:30:09.391 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:30:09.391 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=2 2024-09-17T23:30:09.391 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=2 2024-09-17T23:30:09.391 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-17T23:30:09.408 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-17T23:30:09.408 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-17T23:30:09.408 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-17T23:30:09.408 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_42 2024-09-17T23:30:09.528 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ return 0 2024-09-17T23:30:09.528 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ status_result=' "state": "Replaying"' 2024-09-17T23:30:09.529 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ echo ' "state": "Replaying"' 2024-09-17T23:30:09.529 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep -i Replaying 2024-09-17T23:30:09.529 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: "state": "Replaying" 2024-09-17T23:30:09.531 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_state=started 2024-09-17T23:30:09.531 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test started = started 2024-09-17T23:30:09.531 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:30:09.531 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_42 2024-09-17T23:30:09.531 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:30:09.531 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:30:09.531 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:30:09.531 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_42 2024-09-17T23:30:09.531 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = journal ']' 2024-09-17T23:30:09.531 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = snapshot ']' 2024-09-17T23:30:09.532 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_42 2024-09-17T23:30:09.532 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:30:09.532 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:30:09.532 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:30:09.532 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_42 2024-09-17T23:30:09.532 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster2-mirror-image_42.status 2024-09-17T23:30:09.532 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster2-mirror-image_42.status 2024-09-17T23:30:09.532 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:30:09.533 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_42.status 2024-09-17T23:30:09.533 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster1-mirror-image_42.status 2024-09-17T23:30:09.533 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster1-mirror-image_42.status 2024-09-17T23:30:09.533 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:30:09.533 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_status_log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_42.status 2024-09-17T23:30:09.534 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ mirror_image_snapshot cluster2 mirror image_42 2024-09-17T23:30:09.534 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:30:09.534 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:30:09.534 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_42 2024-09-17T23:30:09.534 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_42 2024-09-17T23:30:10.184 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Snapshot ID: 168 2024-09-17T23:30:10.196 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_42 /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_42.status 2024-09-17T23:30:10.196 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:30:10.196 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:30:10.196 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_42 2024-09-17T23:30:10.197 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_42.status 2024-09-17T23:30:10.197 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_42 --format xml 2024-09-17T23:30:10.197 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:30:10.243 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-17T23:30:10.245 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snapshot_id=168 2024-09-17T23:30:10.245 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:30:10.245 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:30:10.245 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.2 2024-09-17T23:30:10.447 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_42 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_42.status 2024-09-17T23:30:10.447 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:30:10.447 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:30:10.447 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_42 2024-09-17T23:30:10.447 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_42.status 2024-09-17T23:30:10.447 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_42 --format xml 2024-09-17T23:30:10.447 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:30:10.494 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:30:10.494 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:30:10.496 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:30:10.496 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:30:10.496 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:30:10.496 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:30:10.497 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 168 = '' 2024-09-17T23:30:10.497 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:30:10.497 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.4 2024-09-17T23:30:10.897 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_42 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_42.status 2024-09-17T23:30:10.898 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:30:10.898 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:30:10.898 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_42 2024-09-17T23:30:10.898 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_42.status 2024-09-17T23:30:10.898 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_42 --format xml 2024-09-17T23:30:10.898 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:30:10.946 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:30:10.947 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:30:10.948 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:30:10.948 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:30:10.948 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:30:10.949 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:30:10.949 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 168 = '' 2024-09-17T23:30:10.949 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:30:10.949 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.8 2024-09-17T23:30:11.750 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_42 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_42.status 2024-09-17T23:30:11.750 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:30:11.750 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:30:11.750 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_42 2024-09-17T23:30:11.751 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_42.status 2024-09-17T23:30:11.751 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_42 --format xml 2024-09-17T23:30:11.751 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:30:11.799 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:30:11.801 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id=168 2024-09-17T23:30:11.801 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 168 = 168 2024-09-17T23:30:11.801 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:30:11.801 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snap_present cluster1 mirror image_42 snap 2024-09-17T23:30:11.801 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:30:11.801 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:30:11.801 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_42 2024-09-17T23:30:11.801 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:30:11.801 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:30:11.801 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-17T23:30:11.802 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:30:12.802 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 info mirror/image_42@snap 2024-09-17T23:30:12.851 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'image_42': 2024-09-17T23:30:12.851 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 128 MiB in 32 objects 2024-09-17T23:30:12.852 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:30:12.852 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 2 2024-09-17T23:30:12.852 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 1210a2e24a52 2024-09-17T23:30:12.852 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.1210a2e24a52 2024-09-17T23:30:12.852 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:30:12.852 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-17T23:30:12.852 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:30:12.852 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:30:12.852 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:23:35 2024 2024-09-17T23:30:12.852 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:23:35 2024 2024-09-17T23:30:12.852 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:23:35 2024 2024-09-17T23:30:12.852 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: protected: False 2024-09-17T23:30:12.852 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:30:12.852 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:30:12.853 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: 515d2514-6060-40dc-8860-1c0b3ca01bb9 2024-09-17T23:30:12.853 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: false 2024-09-17T23:30:12.855 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:30:12.855 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ compare_image_snaps mirror image_42 snap 2024-09-17T23:30:12.855 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:30:12.855 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_42 2024-09-17T23:30:12.855 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:30:12.856 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local ret=0 2024-09-17T23:30:12.856 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local rmt_export=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_42.export 2024-09-17T23:30:12.856 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local loc_export=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_42.export 2024-09-17T23:30:12.856 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_42.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_42.export 2024-09-17T23:30:12.856 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror export image_42@snap /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_42.export 2024-09-17T23:30:13.021 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-17T23:30:13.027 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 -p mirror export image_42@snap /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_42.export 2024-09-17T23:30:13.256 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-17T23:30:13.264 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ cmp /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_42.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_42.export 2024-09-17T23:30:13.326 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_42.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_42.export 2024-09-17T23:30:13.358 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:30:13.358 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:30:13.358 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_43 2024-09-17T23:30:13.358 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_snap cluster2 mirror image_43 snap 2024-09-17T23:30:13.359 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:30:13.359 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:30:13.359 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_43 2024-09-17T23:30:13.359 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:30:13.359 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror snap create image_43@snap --debug-rbd=20 --debug-journaler=20 2024-09-17T23:30:15.241 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_started cluster1 mirror image_43 2024-09-17T23:30:15.242 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:30:15.242 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:30:15.242 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_43 2024-09-17T23:30:15.242 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_state cluster1 mirror image_43 started 2024-09-17T23:30:15.242 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:30:15.242 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:30:15.242 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_43 2024-09-17T23:30:15.242 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=started 2024-09-17T23:30:15.242 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:30:15.243 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:30:15.243 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:30:16.238 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_replay_state cluster1 mirror image_43 started 2024-09-17T23:30:16.238 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:30:16.239 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:30:16.239 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_43 2024-09-17T23:30:16.239 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=started 2024-09-17T23:30:16.239 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_result 2024-09-17T23:30:16.239 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=stopped 2024-09-17T23:30:16.240 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_43 2024-09-17T23:30:16.240 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_instance=cluster1 2024-09-17T23:30:16.240 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:30:16.240 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ grep -i state 2024-09-17T23:30:16.240 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:30:16.240 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance=cluster1 2024-09-17T23:30:16.240 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local loop_instance 2024-09-17T23:30:16.240 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:30:16.241 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sleep 0 2024-09-17T23:30:16.241 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-17T23:30:16.241 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ seq 0 3 2024-09-17T23:30:16.243 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:30:16.243 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_43 2024-09-17T23:30:16.243 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:0 2024-09-17T23:30:16.243 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:30:16.243 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:30:16.243 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-17T23:30:16.243 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:0 2024-09-17T23:30:16.243 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:30:16.243 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:30:16.243 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:30:16.243 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=0 2024-09-17T23:30:16.244 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-17T23:30:16.244 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:30:16.244 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:30:16.244 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=0 2024-09-17T23:30:16.244 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:30:16.244 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-17T23:30:16.244 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:0 2024-09-17T23:30:16.244 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:30:16.244 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:30:16.244 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-17T23:30:16.244 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:0 2024-09-17T23:30:16.246 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:30:16.246 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:30:16.246 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:30:16.246 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:30:16.246 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-17T23:30:16.246 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:30:16.246 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:30:16.246 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=0 2024-09-17T23:30:16.246 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=0 2024-09-17T23:30:16.246 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-17T23:30:16.265 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:30:16.265 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:30:16.266 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:30:16.266 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_43 2024-09-17T23:30:16.348 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:Invalid command: unused arguments: ['mirror/image_43'] 2024-09-17T23:30:16.348 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd mirror status : get status for rbd mirror 2024-09-17T23:30:16.348 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:admin_socket: invalid command 2024-09-17T23:30:16.351 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:30:16.351 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_43 2024-09-17T23:30:16.351 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:1 2024-09-17T23:30:16.351 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:30:16.351 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:30:16.351 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-17T23:30:16.351 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:1 2024-09-17T23:30:16.351 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:30:16.352 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:30:16.352 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:30:16.352 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=1 2024-09-17T23:30:16.352 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-17T23:30:16.352 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:30:16.352 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:30:16.352 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=1 2024-09-17T23:30:16.352 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=1 2024-09-17T23:30:16.352 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-17T23:30:16.352 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:1 2024-09-17T23:30:16.352 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:30:16.352 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:30:16.352 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-17T23:30:16.352 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:1 2024-09-17T23:30:16.353 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:30:16.353 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:30:16.353 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:30:16.353 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=1 2024-09-17T23:30:16.353 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-17T23:30:16.353 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:30:16.353 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:30:16.353 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=1 2024-09-17T23:30:16.353 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=1 2024-09-17T23:30:16.353 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-17T23:30:16.371 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-17T23:30:16.371 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-17T23:30:16.371 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-17T23:30:16.371 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_43 2024-09-17T23:30:16.447 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:Invalid command: unused arguments: ['mirror/image_43'] 2024-09-17T23:30:16.447 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd mirror status : get status for rbd mirror 2024-09-17T23:30:16.447 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:admin_socket: invalid command 2024-09-17T23:30:16.449 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:30:16.450 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_43 2024-09-17T23:30:16.450 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:2 2024-09-17T23:30:16.450 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:30:16.450 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:30:16.450 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-17T23:30:16.450 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:2 2024-09-17T23:30:16.450 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:30:16.450 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:30:16.450 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:30:16.450 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=2 2024-09-17T23:30:16.450 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-17T23:30:16.450 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:30:16.450 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:30:16.451 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=2 2024-09-17T23:30:16.451 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=2 2024-09-17T23:30:16.451 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-17T23:30:16.451 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:2 2024-09-17T23:30:16.451 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:30:16.451 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:30:16.451 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-17T23:30:16.451 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:2 2024-09-17T23:30:16.451 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:30:16.451 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:30:16.451 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:30:16.451 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=2 2024-09-17T23:30:16.451 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-17T23:30:16.451 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:30:16.451 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:30:16.452 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=2 2024-09-17T23:30:16.452 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=2 2024-09-17T23:30:16.452 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-17T23:30:16.469 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-17T23:30:16.469 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-17T23:30:16.469 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-17T23:30:16.470 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_43 2024-09-17T23:30:16.589 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ return 0 2024-09-17T23:30:16.590 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ status_result=' "state": "Replaying"' 2024-09-17T23:30:16.590 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ echo ' "state": "Replaying"' 2024-09-17T23:30:16.590 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep -i Replaying 2024-09-17T23:30:16.590 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: "state": "Replaying" 2024-09-17T23:30:16.592 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_state=started 2024-09-17T23:30:16.592 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test started = started 2024-09-17T23:30:16.592 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:30:16.592 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_43 2024-09-17T23:30:16.592 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:30:16.593 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:30:16.593 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:30:16.593 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_43 2024-09-17T23:30:16.593 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = journal ']' 2024-09-17T23:30:16.593 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = snapshot ']' 2024-09-17T23:30:16.593 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_43 2024-09-17T23:30:16.593 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:30:16.593 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:30:16.593 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:30:16.593 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_43 2024-09-17T23:30:16.593 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster2-mirror-image_43.status 2024-09-17T23:30:16.593 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster2-mirror-image_43.status 2024-09-17T23:30:16.594 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:30:16.595 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_43.status 2024-09-17T23:30:16.595 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster1-mirror-image_43.status 2024-09-17T23:30:16.595 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster1-mirror-image_43.status 2024-09-17T23:30:16.595 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:30:16.595 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_status_log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_43.status 2024-09-17T23:30:16.595 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ mirror_image_snapshot cluster2 mirror image_43 2024-09-17T23:30:16.595 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:30:16.595 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:30:16.595 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_43 2024-09-17T23:30:16.596 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_43 2024-09-17T23:30:17.232 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Snapshot ID: 170 2024-09-17T23:30:17.244 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_43 /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_43.status 2024-09-17T23:30:17.244 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:30:17.244 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:30:17.244 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_43 2024-09-17T23:30:17.245 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_43.status 2024-09-17T23:30:17.245 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_43 --format xml 2024-09-17T23:30:17.245 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:30:17.289 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-17T23:30:17.291 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snapshot_id=170 2024-09-17T23:30:17.291 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:30:17.291 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:30:17.291 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.2 2024-09-17T23:30:17.493 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_43 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_43.status 2024-09-17T23:30:17.493 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:30:17.493 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:30:17.493 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_43 2024-09-17T23:30:17.494 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_43.status 2024-09-17T23:30:17.494 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_43 --format xml 2024-09-17T23:30:17.494 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:30:17.540 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:30:17.540 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:30:17.542 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:30:17.542 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:30:17.542 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:30:17.542 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:30:17.542 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 170 = '' 2024-09-17T23:30:17.542 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:30:17.542 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.4 2024-09-17T23:30:17.943 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_43 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_43.status 2024-09-17T23:30:17.944 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:30:17.944 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:30:17.944 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_43 2024-09-17T23:30:17.944 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_43.status 2024-09-17T23:30:17.944 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_43 --format xml 2024-09-17T23:30:17.944 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:30:17.991 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:30:17.992 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:30:17.993 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:30:17.993 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:30:17.993 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:30:17.993 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:30:17.994 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 170 = '' 2024-09-17T23:30:17.994 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:30:17.994 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.8 2024-09-17T23:30:18.795 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_43 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_43.status 2024-09-17T23:30:18.795 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:30:18.795 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:30:18.795 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_43 2024-09-17T23:30:18.795 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_43.status 2024-09-17T23:30:18.796 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_43 --format xml 2024-09-17T23:30:18.796 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:30:18.844 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:30:18.846 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id=170 2024-09-17T23:30:18.846 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 170 = 170 2024-09-17T23:30:18.847 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:30:18.847 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snap_present cluster1 mirror image_43 snap 2024-09-17T23:30:18.847 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:30:18.847 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:30:18.847 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_43 2024-09-17T23:30:18.847 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:30:18.847 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:30:18.847 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-17T23:30:18.847 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:30:19.847 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 info mirror/image_43@snap 2024-09-17T23:30:19.897 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'image_43': 2024-09-17T23:30:19.898 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 128 MiB in 32 objects 2024-09-17T23:30:19.898 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:30:19.898 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 2 2024-09-17T23:30:19.898 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 121090e9b2fc 2024-09-17T23:30:19.898 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.121090e9b2fc 2024-09-17T23:30:19.898 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:30:19.898 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-17T23:30:19.898 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:30:19.898 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:30:19.898 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:23:49 2024 2024-09-17T23:30:19.898 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:23:49 2024 2024-09-17T23:30:19.898 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:23:49 2024 2024-09-17T23:30:19.899 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: protected: False 2024-09-17T23:30:19.899 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:30:19.899 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:30:19.899 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: 65d764ca-56da-405c-b68b-d13fb878db55 2024-09-17T23:30:19.899 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: false 2024-09-17T23:30:19.901 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:30:19.901 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ compare_image_snaps mirror image_43 snap 2024-09-17T23:30:19.901 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:30:19.902 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_43 2024-09-17T23:30:19.902 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:30:19.902 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local ret=0 2024-09-17T23:30:19.902 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local rmt_export=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_43.export 2024-09-17T23:30:19.902 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local loc_export=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_43.export 2024-09-17T23:30:19.902 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_43.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_43.export 2024-09-17T23:30:19.902 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror export image_43@snap /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_43.export 2024-09-17T23:30:20.064 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-17T23:30:20.071 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 -p mirror export image_43@snap /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_43.export 2024-09-17T23:30:20.288 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-17T23:30:20.294 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ cmp /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_43.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_43.export 2024-09-17T23:30:20.361 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_43.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_43.export 2024-09-17T23:30:20.390 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:30:20.390 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:30:20.390 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_44 2024-09-17T23:30:20.391 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_snap cluster2 mirror image_44 snap 2024-09-17T23:30:20.391 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:30:20.391 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:30:20.391 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_44 2024-09-17T23:30:20.391 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:30:20.391 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror snap create image_44@snap --debug-rbd=20 --debug-journaler=20 2024-09-17T23:30:22.252 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_started cluster1 mirror image_44 2024-09-17T23:30:22.253 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:30:22.253 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:30:22.253 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_44 2024-09-17T23:30:22.253 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_state cluster1 mirror image_44 started 2024-09-17T23:30:22.253 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:30:22.253 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:30:22.253 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_44 2024-09-17T23:30:22.253 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=started 2024-09-17T23:30:22.253 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:30:22.253 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:30:22.254 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:30:23.253 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_replay_state cluster1 mirror image_44 started 2024-09-17T23:30:23.253 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:30:23.254 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:30:23.254 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_44 2024-09-17T23:30:23.254 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=started 2024-09-17T23:30:23.254 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_result 2024-09-17T23:30:23.254 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=stopped 2024-09-17T23:30:23.255 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_44 2024-09-17T23:30:23.255 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ grep -i state 2024-09-17T23:30:23.255 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_instance=cluster1 2024-09-17T23:30:23.255 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:30:23.256 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:30:23.256 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance=cluster1 2024-09-17T23:30:23.256 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local loop_instance 2024-09-17T23:30:23.256 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:30:23.256 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sleep 0 2024-09-17T23:30:23.256 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-17T23:30:23.256 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ seq 0 3 2024-09-17T23:30:23.258 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:30:23.258 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_44 2024-09-17T23:30:23.258 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:0 2024-09-17T23:30:23.258 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:30:23.258 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:30:23.258 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-17T23:30:23.258 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:0 2024-09-17T23:30:23.258 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:30:23.259 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:30:23.259 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:30:23.259 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=0 2024-09-17T23:30:23.259 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-17T23:30:23.259 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:30:23.259 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:30:23.259 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=0 2024-09-17T23:30:23.259 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:30:23.260 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-17T23:30:23.260 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:0 2024-09-17T23:30:23.260 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:30:23.260 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:30:23.260 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-17T23:30:23.263 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:0 2024-09-17T23:30:23.263 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:30:23.263 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:30:23.263 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:30:23.263 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:30:23.263 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-17T23:30:23.263 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:30:23.263 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:30:23.263 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=0 2024-09-17T23:30:23.263 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=0 2024-09-17T23:30:23.263 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-17T23:30:23.282 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:30:23.282 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:30:23.282 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:30:23.282 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_44 2024-09-17T23:30:23.364 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:Invalid command: unused arguments: ['mirror/image_44'] 2024-09-17T23:30:23.364 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd mirror status : get status for rbd mirror 2024-09-17T23:30:23.364 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:admin_socket: invalid command 2024-09-17T23:30:23.367 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:30:23.368 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_44 2024-09-17T23:30:23.368 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:1 2024-09-17T23:30:23.368 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:30:23.368 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:30:23.368 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-17T23:30:23.368 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:1 2024-09-17T23:30:23.368 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:30:23.368 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:30:23.368 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:30:23.368 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=1 2024-09-17T23:30:23.368 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-17T23:30:23.368 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:30:23.368 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:30:23.368 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=1 2024-09-17T23:30:23.369 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=1 2024-09-17T23:30:23.369 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-17T23:30:23.369 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:1 2024-09-17T23:30:23.369 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:30:23.369 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:30:23.369 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-17T23:30:23.369 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:1 2024-09-17T23:30:23.369 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:30:23.369 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:30:23.369 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:30:23.369 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=1 2024-09-17T23:30:23.369 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-17T23:30:23.369 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:30:23.369 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:30:23.370 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=1 2024-09-17T23:30:23.370 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=1 2024-09-17T23:30:23.370 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-17T23:30:23.386 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-17T23:30:23.386 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-17T23:30:23.386 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-17T23:30:23.386 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_44 2024-09-17T23:30:23.463 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:Invalid command: unused arguments: ['mirror/image_44'] 2024-09-17T23:30:23.463 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd mirror status : get status for rbd mirror 2024-09-17T23:30:23.463 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:admin_socket: invalid command 2024-09-17T23:30:23.466 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:30:23.466 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_44 2024-09-17T23:30:23.466 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:2 2024-09-17T23:30:23.467 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:30:23.467 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:30:23.467 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-17T23:30:23.467 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:2 2024-09-17T23:30:23.467 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:30:23.467 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:30:23.467 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:30:23.467 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=2 2024-09-17T23:30:23.467 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-17T23:30:23.467 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:30:23.467 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:30:23.467 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=2 2024-09-17T23:30:23.467 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=2 2024-09-17T23:30:23.468 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-17T23:30:23.468 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:2 2024-09-17T23:30:23.468 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:30:23.468 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:30:23.468 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-17T23:30:23.468 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:2 2024-09-17T23:30:23.468 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:30:23.468 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:30:23.468 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:30:23.469 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=2 2024-09-17T23:30:23.469 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-17T23:30:23.469 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:30:23.469 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:30:23.469 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=2 2024-09-17T23:30:23.469 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=2 2024-09-17T23:30:23.469 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-17T23:30:23.486 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-17T23:30:23.486 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-17T23:30:23.486 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-17T23:30:23.486 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_44 2024-09-17T23:30:23.605 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ return 0 2024-09-17T23:30:23.606 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ status_result=' "state": "Replaying"' 2024-09-17T23:30:23.606 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ echo ' "state": "Replaying"' 2024-09-17T23:30:23.606 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep -i Replaying 2024-09-17T23:30:23.606 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: "state": "Replaying" 2024-09-17T23:30:23.608 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_state=started 2024-09-17T23:30:23.608 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test started = started 2024-09-17T23:30:23.608 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:30:23.608 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_44 2024-09-17T23:30:23.608 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:30:23.608 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:30:23.608 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:30:23.609 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_44 2024-09-17T23:30:23.609 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = journal ']' 2024-09-17T23:30:23.609 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = snapshot ']' 2024-09-17T23:30:23.609 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_44 2024-09-17T23:30:23.609 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:30:23.609 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:30:23.609 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:30:23.609 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_44 2024-09-17T23:30:23.609 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster2-mirror-image_44.status 2024-09-17T23:30:23.609 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster2-mirror-image_44.status 2024-09-17T23:30:23.609 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:30:23.611 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_44.status 2024-09-17T23:30:23.611 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster1-mirror-image_44.status 2024-09-17T23:30:23.611 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster1-mirror-image_44.status 2024-09-17T23:30:23.611 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:30:23.611 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_status_log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_44.status 2024-09-17T23:30:23.611 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ mirror_image_snapshot cluster2 mirror image_44 2024-09-17T23:30:23.611 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:30:23.612 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:30:23.612 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_44 2024-09-17T23:30:23.612 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_44 2024-09-17T23:30:24.261 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Snapshot ID: 172 2024-09-17T23:30:24.274 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_44 /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_44.status 2024-09-17T23:30:24.274 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:30:24.274 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:30:24.274 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_44 2024-09-17T23:30:24.274 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_44.status 2024-09-17T23:30:24.274 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_44 --format xml 2024-09-17T23:30:24.274 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:30:24.321 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-17T23:30:24.323 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snapshot_id=172 2024-09-17T23:30:24.323 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:30:24.323 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:30:24.323 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.2 2024-09-17T23:30:24.525 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_44 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_44.status 2024-09-17T23:30:24.525 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:30:24.525 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:30:24.525 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_44 2024-09-17T23:30:24.525 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_44.status 2024-09-17T23:30:24.525 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_44 --format xml 2024-09-17T23:30:24.525 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:30:24.578 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:30:24.579 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:30:24.580 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:30:24.580 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:30:24.580 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:30:24.581 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:30:24.581 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 172 = '' 2024-09-17T23:30:24.581 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:30:24.581 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.4 2024-09-17T23:30:24.982 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_44 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_44.status 2024-09-17T23:30:24.982 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:30:24.982 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:30:24.982 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_44 2024-09-17T23:30:24.983 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_44.status 2024-09-17T23:30:24.983 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_44 --format xml 2024-09-17T23:30:24.983 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:30:25.034 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:30:25.034 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:30:25.036 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:30:25.036 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:30:25.036 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:30:25.036 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:30:25.036 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 172 = '' 2024-09-17T23:30:25.036 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:30:25.037 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.8 2024-09-17T23:30:25.838 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_44 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_44.status 2024-09-17T23:30:25.838 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:30:25.838 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:30:25.838 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_44 2024-09-17T23:30:25.838 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_44.status 2024-09-17T23:30:25.838 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_44 --format xml 2024-09-17T23:30:25.838 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:30:25.887 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:30:25.889 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id=172 2024-09-17T23:30:25.892 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 172 = 172 2024-09-17T23:30:25.892 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:30:25.892 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snap_present cluster1 mirror image_44 snap 2024-09-17T23:30:25.892 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:30:25.892 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:30:25.892 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_44 2024-09-17T23:30:25.892 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:30:25.892 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:30:25.892 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-17T23:30:25.893 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:30:26.890 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 info mirror/image_44@snap 2024-09-17T23:30:26.971 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'image_44': 2024-09-17T23:30:26.971 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 128 MiB in 32 objects 2024-09-17T23:30:26.971 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:30:26.971 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 2 2024-09-17T23:30:26.971 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 12107f64e012 2024-09-17T23:30:26.972 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.12107f64e012 2024-09-17T23:30:26.972 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:30:26.972 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-17T23:30:26.972 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:30:26.972 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:30:26.972 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:23:59 2024 2024-09-17T23:30:26.972 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:23:59 2024 2024-09-17T23:30:26.972 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:23:59 2024 2024-09-17T23:30:26.972 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: protected: False 2024-09-17T23:30:26.972 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:30:26.972 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:30:26.972 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: 1c859b2d-e379-40d7-bc46-427fb4a14245 2024-09-17T23:30:26.972 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: false 2024-09-17T23:30:26.976 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:30:26.976 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ compare_image_snaps mirror image_44 snap 2024-09-17T23:30:26.976 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:30:26.976 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_44 2024-09-17T23:30:26.976 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:30:26.976 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local ret=0 2024-09-17T23:30:26.976 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local rmt_export=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_44.export 2024-09-17T23:30:26.976 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local loc_export=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_44.export 2024-09-17T23:30:26.976 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_44.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_44.export 2024-09-17T23:30:26.977 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror export image_44@snap /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_44.export 2024-09-17T23:30:27.142 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-17T23:30:27.151 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 -p mirror export image_44@snap /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_44.export 2024-09-17T23:30:27.376 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-17T23:30:27.383 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ cmp /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_44.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_44.export 2024-09-17T23:30:27.445 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_44.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_44.export 2024-09-17T23:30:27.478 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:30:27.478 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:30:27.478 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_45 2024-09-17T23:30:27.478 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_snap cluster2 mirror image_45 snap 2024-09-17T23:30:27.478 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:30:27.478 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:30:27.479 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_45 2024-09-17T23:30:27.479 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:30:27.479 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror snap create image_45@snap --debug-rbd=20 --debug-journaler=20 2024-09-17T23:30:29.306 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_started cluster1 mirror image_45 2024-09-17T23:30:29.306 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:30:29.306 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:30:29.306 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_45 2024-09-17T23:30:29.306 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_state cluster1 mirror image_45 started 2024-09-17T23:30:29.306 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:30:29.306 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:30:29.306 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_45 2024-09-17T23:30:29.306 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=started 2024-09-17T23:30:29.306 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:30:29.307 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:30:29.307 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:30:30.307 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_replay_state cluster1 mirror image_45 started 2024-09-17T23:30:30.307 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:30:30.307 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:30:30.307 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_45 2024-09-17T23:30:30.307 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=started 2024-09-17T23:30:30.308 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_result 2024-09-17T23:30:30.308 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=stopped 2024-09-17T23:30:30.309 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_45 2024-09-17T23:30:30.309 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_instance=cluster1 2024-09-17T23:30:30.309 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:30:30.309 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ grep -i state 2024-09-17T23:30:30.309 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:30:30.309 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance=cluster1 2024-09-17T23:30:30.309 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local loop_instance 2024-09-17T23:30:30.309 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:30:30.309 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sleep 0 2024-09-17T23:30:30.309 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-17T23:30:30.311 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ seq 0 3 2024-09-17T23:30:30.311 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:30:30.311 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_45 2024-09-17T23:30:30.311 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:0 2024-09-17T23:30:30.311 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:30:30.311 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:30:30.311 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-17T23:30:30.311 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:0 2024-09-17T23:30:30.312 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:30:30.312 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:30:30.312 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:30:30.312 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=0 2024-09-17T23:30:30.312 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-17T23:30:30.312 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:30:30.312 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:30:30.312 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=0 2024-09-17T23:30:30.312 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:30:30.312 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-17T23:30:30.312 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:0 2024-09-17T23:30:30.313 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:30:30.314 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:30:30.314 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-17T23:30:30.314 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:0 2024-09-17T23:30:30.314 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:30:30.314 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:30:30.314 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:30:30.314 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:30:30.314 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-17T23:30:30.314 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:30:30.314 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:30:30.314 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=0 2024-09-17T23:30:30.314 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=0 2024-09-17T23:30:30.314 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-17T23:30:30.335 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:30:30.335 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:30:30.335 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:30:30.335 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_45 2024-09-17T23:30:30.418 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:Invalid command: unused arguments: ['mirror/image_45'] 2024-09-17T23:30:30.418 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd mirror status : get status for rbd mirror 2024-09-17T23:30:30.418 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:admin_socket: invalid command 2024-09-17T23:30:30.421 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:30:30.421 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_45 2024-09-17T23:30:30.421 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:1 2024-09-17T23:30:30.421 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:30:30.421 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:30:30.421 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-17T23:30:30.421 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:1 2024-09-17T23:30:30.421 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:30:30.422 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:30:30.422 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:30:30.422 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=1 2024-09-17T23:30:30.422 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-17T23:30:30.422 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:30:30.422 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:30:30.422 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=1 2024-09-17T23:30:30.422 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=1 2024-09-17T23:30:30.422 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-17T23:30:30.422 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:1 2024-09-17T23:30:30.422 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:30:30.422 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:30:30.423 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-17T23:30:30.423 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:1 2024-09-17T23:30:30.423 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:30:30.423 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:30:30.423 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:30:30.423 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=1 2024-09-17T23:30:30.423 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-17T23:30:30.423 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:30:30.423 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:30:30.423 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=1 2024-09-17T23:30:30.423 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=1 2024-09-17T23:30:30.423 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-17T23:30:30.440 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-17T23:30:30.441 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-17T23:30:30.441 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-17T23:30:30.441 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_45 2024-09-17T23:30:30.517 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:Invalid command: unused arguments: ['mirror/image_45'] 2024-09-17T23:30:30.517 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd mirror status : get status for rbd mirror 2024-09-17T23:30:30.517 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:admin_socket: invalid command 2024-09-17T23:30:30.520 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:30:30.520 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_45 2024-09-17T23:30:30.520 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:2 2024-09-17T23:30:30.520 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:30:30.520 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:30:30.520 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-17T23:30:30.520 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:2 2024-09-17T23:30:30.520 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:30:30.520 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:30:30.520 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:30:30.521 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=2 2024-09-17T23:30:30.521 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-17T23:30:30.521 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:30:30.521 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:30:30.521 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=2 2024-09-17T23:30:30.521 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=2 2024-09-17T23:30:30.521 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-17T23:30:30.521 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:2 2024-09-17T23:30:30.521 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:30:30.521 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:30:30.521 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-17T23:30:30.521 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:2 2024-09-17T23:30:30.521 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:30:30.521 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:30:30.522 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:30:30.522 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=2 2024-09-17T23:30:30.522 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-17T23:30:30.522 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:30:30.522 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:30:30.522 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=2 2024-09-17T23:30:30.523 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=2 2024-09-17T23:30:30.523 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-17T23:30:30.539 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-17T23:30:30.539 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-17T23:30:30.539 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-17T23:30:30.539 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_45 2024-09-17T23:30:30.658 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ return 0 2024-09-17T23:30:30.659 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ status_result=' "state": "Replaying"' 2024-09-17T23:30:30.659 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ echo ' "state": "Replaying"' 2024-09-17T23:30:30.659 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep -i Replaying 2024-09-17T23:30:30.659 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: "state": "Replaying" 2024-09-17T23:30:30.661 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_state=started 2024-09-17T23:30:30.661 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test started = started 2024-09-17T23:30:30.661 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:30:30.661 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_45 2024-09-17T23:30:30.661 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:30:30.661 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:30:30.661 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:30:30.661 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_45 2024-09-17T23:30:30.661 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = journal ']' 2024-09-17T23:30:30.661 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = snapshot ']' 2024-09-17T23:30:30.662 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_45 2024-09-17T23:30:30.662 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:30:30.662 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:30:30.662 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:30:30.662 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_45 2024-09-17T23:30:30.662 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster2-mirror-image_45.status 2024-09-17T23:30:30.662 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster2-mirror-image_45.status 2024-09-17T23:30:30.662 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:30:30.663 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_45.status 2024-09-17T23:30:30.663 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster1-mirror-image_45.status 2024-09-17T23:30:30.663 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster1-mirror-image_45.status 2024-09-17T23:30:30.663 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:30:30.664 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_status_log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_45.status 2024-09-17T23:30:30.664 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ mirror_image_snapshot cluster2 mirror image_45 2024-09-17T23:30:30.664 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:30:30.664 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:30:30.664 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_45 2024-09-17T23:30:30.664 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_45 2024-09-17T23:30:31.306 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Snapshot ID: 174 2024-09-17T23:30:31.512 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_45 /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_45.status 2024-09-17T23:30:31.512 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:30:31.512 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:30:31.512 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_45 2024-09-17T23:30:31.512 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_45.status 2024-09-17T23:30:31.513 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_45 --format xml 2024-09-17T23:30:31.513 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:30:31.513 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-17T23:30:31.513 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snapshot_id=174 2024-09-17T23:30:31.513 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:30:31.513 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:30:31.513 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.2 2024-09-17T23:30:31.570 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_45 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_45.status 2024-09-17T23:30:31.570 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:30:31.570 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:30:31.570 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_45 2024-09-17T23:30:31.570 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_45.status 2024-09-17T23:30:31.570 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_45 --format xml 2024-09-17T23:30:31.570 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:30:31.623 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:30:31.625 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id=79 2024-09-17T23:30:31.625 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 174 = 79 2024-09-17T23:30:31.625 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:30:31.625 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.4 2024-09-17T23:30:32.027 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_45 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_45.status 2024-09-17T23:30:32.027 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:30:32.027 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:30:32.027 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_45 2024-09-17T23:30:32.027 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_45.status 2024-09-17T23:30:32.027 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_45 --format xml 2024-09-17T23:30:32.027 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:30:32.081 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:30:32.083 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id=79 2024-09-17T23:30:32.083 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 174 = 79 2024-09-17T23:30:32.083 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:30:32.083 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.8 2024-09-17T23:30:32.884 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_45 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_45.status 2024-09-17T23:30:32.884 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:30:32.884 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:30:32.884 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_45 2024-09-17T23:30:32.884 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_45.status 2024-09-17T23:30:32.884 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_45 --format xml 2024-09-17T23:30:32.884 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:30:32.934 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:30:32.934 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:30:32.936 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:30:32.936 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:30:32.936 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:30:32.937 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:30:32.937 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 174 = '' 2024-09-17T23:30:32.937 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:30:32.937 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1.6 2024-09-17T23:30:34.538 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_45 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_45.status 2024-09-17T23:30:34.538 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:30:34.538 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:30:34.538 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_45 2024-09-17T23:30:34.538 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_45.status 2024-09-17T23:30:34.538 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_45 --format xml 2024-09-17T23:30:34.538 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:30:34.590 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:30:34.592 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id=174 2024-09-17T23:30:34.592 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 174 = 174 2024-09-17T23:30:34.592 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:30:34.593 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snap_present cluster1 mirror image_45 snap 2024-09-17T23:30:34.593 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:30:34.593 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:30:34.593 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_45 2024-09-17T23:30:34.593 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:30:34.593 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:30:34.593 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-17T23:30:34.593 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:30:35.593 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 info mirror/image_45@snap 2024-09-17T23:30:35.643 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'image_45': 2024-09-17T23:30:35.643 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 128 MiB in 32 objects 2024-09-17T23:30:35.643 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:30:35.643 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 2 2024-09-17T23:30:35.643 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 1210acf7e455 2024-09-17T23:30:35.643 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.1210acf7e455 2024-09-17T23:30:35.643 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:30:35.643 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-17T23:30:35.643 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:30:35.643 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:30:35.643 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:24:06 2024 2024-09-17T23:30:35.644 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:24:06 2024 2024-09-17T23:30:35.644 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:24:06 2024 2024-09-17T23:30:35.644 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: protected: False 2024-09-17T23:30:35.644 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:30:35.644 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:30:35.644 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: a44d8381-af32-4227-90fa-0531c242a7f7 2024-09-17T23:30:35.644 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: false 2024-09-17T23:30:35.646 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:30:35.646 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ compare_image_snaps mirror image_45 snap 2024-09-17T23:30:35.646 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:30:35.646 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_45 2024-09-17T23:30:35.647 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:30:35.647 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local ret=0 2024-09-17T23:30:35.647 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local rmt_export=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_45.export 2024-09-17T23:30:35.647 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local loc_export=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_45.export 2024-09-17T23:30:35.647 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_45.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_45.export 2024-09-17T23:30:35.647 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror export image_45@snap /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_45.export 2024-09-17T23:30:35.818 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-17T23:30:35.825 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 -p mirror export image_45@snap /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_45.export 2024-09-17T23:30:36.048 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-17T23:30:36.056 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ cmp /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_45.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_45.export 2024-09-17T23:30:36.118 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_45.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_45.export 2024-09-17T23:30:36.151 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:30:36.151 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:30:36.151 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_46 2024-09-17T23:30:36.151 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_snap cluster2 mirror image_46 snap 2024-09-17T23:30:36.151 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:30:36.151 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:30:36.151 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_46 2024-09-17T23:30:36.151 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:30:36.151 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror snap create image_46@snap --debug-rbd=20 --debug-journaler=20 2024-09-17T23:30:38.001 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_started cluster1 mirror image_46 2024-09-17T23:30:38.001 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:30:38.002 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:30:38.002 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_46 2024-09-17T23:30:38.002 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_state cluster1 mirror image_46 started 2024-09-17T23:30:38.002 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:30:38.002 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:30:38.002 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_46 2024-09-17T23:30:38.002 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=started 2024-09-17T23:30:38.002 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:30:38.002 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:30:38.002 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:30:39.003 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_replay_state cluster1 mirror image_46 started 2024-09-17T23:30:39.003 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:30:39.003 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:30:39.003 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_46 2024-09-17T23:30:39.003 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=started 2024-09-17T23:30:39.003 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_result 2024-09-17T23:30:39.003 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=stopped 2024-09-17T23:30:39.004 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_46 2024-09-17T23:30:39.004 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_instance=cluster1 2024-09-17T23:30:39.004 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:30:39.004 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ grep -i state 2024-09-17T23:30:39.005 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:30:39.005 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance=cluster1 2024-09-17T23:30:39.005 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local loop_instance 2024-09-17T23:30:39.005 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:30:39.005 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sleep 0 2024-09-17T23:30:39.005 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-17T23:30:39.007 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ seq 0 3 2024-09-17T23:30:39.007 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:30:39.007 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_46 2024-09-17T23:30:39.007 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:0 2024-09-17T23:30:39.007 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:30:39.007 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:30:39.007 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-17T23:30:39.007 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:0 2024-09-17T23:30:39.007 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:30:39.008 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:30:39.008 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:30:39.008 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=0 2024-09-17T23:30:39.008 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-17T23:30:39.008 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:30:39.008 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:30:39.008 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=0 2024-09-17T23:30:39.008 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:30:39.008 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-17T23:30:39.008 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:0 2024-09-17T23:30:39.008 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:30:39.008 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:30:39.008 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-17T23:30:39.008 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:0 2024-09-17T23:30:39.009 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:30:39.009 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:30:39.010 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:30:39.010 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:30:39.010 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-17T23:30:39.010 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:30:39.010 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:30:39.010 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=0 2024-09-17T23:30:39.010 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=0 2024-09-17T23:30:39.010 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-17T23:30:39.030 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:30:39.030 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:30:39.030 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:30:39.030 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_46 2024-09-17T23:30:39.114 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:Invalid command: unused arguments: ['mirror/image_46'] 2024-09-17T23:30:39.114 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd mirror status : get status for rbd mirror 2024-09-17T23:30:39.114 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:admin_socket: invalid command 2024-09-17T23:30:39.117 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:30:39.117 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_46 2024-09-17T23:30:39.117 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:1 2024-09-17T23:30:39.117 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:30:39.117 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:30:39.117 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-17T23:30:39.118 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:1 2024-09-17T23:30:39.118 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:30:39.118 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:30:39.118 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:30:39.118 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=1 2024-09-17T23:30:39.118 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-17T23:30:39.118 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:30:39.118 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:30:39.118 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=1 2024-09-17T23:30:39.118 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=1 2024-09-17T23:30:39.118 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-17T23:30:39.118 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:1 2024-09-17T23:30:39.118 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:30:39.118 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:30:39.119 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-17T23:30:39.119 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:1 2024-09-17T23:30:39.119 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:30:39.119 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:30:39.119 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:30:39.119 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=1 2024-09-17T23:30:39.119 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-17T23:30:39.119 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:30:39.119 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:30:39.119 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=1 2024-09-17T23:30:39.119 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=1 2024-09-17T23:30:39.119 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-17T23:30:39.137 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-17T23:30:39.137 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-17T23:30:39.137 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-17T23:30:39.137 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_46 2024-09-17T23:30:39.213 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:Invalid command: unused arguments: ['mirror/image_46'] 2024-09-17T23:30:39.213 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd mirror status : get status for rbd mirror 2024-09-17T23:30:39.213 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:admin_socket: invalid command 2024-09-17T23:30:39.216 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:30:39.216 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_46 2024-09-17T23:30:39.216 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:2 2024-09-17T23:30:39.216 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:30:39.216 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:30:39.216 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-17T23:30:39.216 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:2 2024-09-17T23:30:39.216 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:30:39.216 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:30:39.217 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:30:39.217 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=2 2024-09-17T23:30:39.217 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-17T23:30:39.217 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:30:39.217 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:30:39.217 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=2 2024-09-17T23:30:39.217 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=2 2024-09-17T23:30:39.217 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-17T23:30:39.217 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:2 2024-09-17T23:30:39.217 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:30:39.217 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:30:39.217 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-17T23:30:39.217 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:2 2024-09-17T23:30:39.217 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:30:39.217 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:30:39.218 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:30:39.218 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=2 2024-09-17T23:30:39.218 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-17T23:30:39.218 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:30:39.218 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:30:39.218 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=2 2024-09-17T23:30:39.218 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=2 2024-09-17T23:30:39.218 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-17T23:30:39.235 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-17T23:30:39.235 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-17T23:30:39.235 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-17T23:30:39.235 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_46 2024-09-17T23:30:39.354 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ return 0 2024-09-17T23:30:39.354 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ status_result=' "state": "Replaying"' 2024-09-17T23:30:39.355 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ echo ' "state": "Replaying"' 2024-09-17T23:30:39.355 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep -i Replaying 2024-09-17T23:30:39.355 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: "state": "Replaying" 2024-09-17T23:30:39.357 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_state=started 2024-09-17T23:30:39.357 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test started = started 2024-09-17T23:30:39.357 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:30:39.357 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_46 2024-09-17T23:30:39.357 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:30:39.357 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:30:39.357 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:30:39.357 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_46 2024-09-17T23:30:39.357 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = journal ']' 2024-09-17T23:30:39.358 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = snapshot ']' 2024-09-17T23:30:39.358 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_46 2024-09-17T23:30:39.358 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:30:39.358 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:30:39.358 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:30:39.358 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_46 2024-09-17T23:30:39.358 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster2-mirror-image_46.status 2024-09-17T23:30:39.358 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster2-mirror-image_46.status 2024-09-17T23:30:39.358 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:30:39.359 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_46.status 2024-09-17T23:30:39.359 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster1-mirror-image_46.status 2024-09-17T23:30:39.359 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster1-mirror-image_46.status 2024-09-17T23:30:39.359 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:30:39.360 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_status_log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_46.status 2024-09-17T23:30:39.360 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ mirror_image_snapshot cluster2 mirror image_46 2024-09-17T23:30:39.360 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:30:39.360 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:30:39.360 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_46 2024-09-17T23:30:39.360 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_46 2024-09-17T23:30:40.017 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Snapshot ID: 176 2024-09-17T23:30:40.031 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_46 /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_46.status 2024-09-17T23:30:40.031 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:30:40.031 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:30:40.031 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_46 2024-09-17T23:30:40.031 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_46.status 2024-09-17T23:30:40.031 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_46 --format xml 2024-09-17T23:30:40.031 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:30:40.078 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-17T23:30:40.080 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snapshot_id=176 2024-09-17T23:30:40.080 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:30:40.080 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:30:40.080 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.2 2024-09-17T23:30:40.281 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_46 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_46.status 2024-09-17T23:30:40.282 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:30:40.282 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:30:40.282 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_46 2024-09-17T23:30:40.282 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_46.status 2024-09-17T23:30:40.282 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_46 --format xml 2024-09-17T23:30:40.282 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:30:40.332 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:30:40.332 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:30:40.334 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:30:40.334 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:30:40.334 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:30:40.334 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:30:40.334 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 176 = '' 2024-09-17T23:30:40.335 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:30:40.335 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.4 2024-09-17T23:30:40.736 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_46 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_46.status 2024-09-17T23:30:40.736 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:30:40.736 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:30:40.736 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_46 2024-09-17T23:30:40.736 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_46.status 2024-09-17T23:30:40.736 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_46 --format xml 2024-09-17T23:30:40.736 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:30:40.789 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:30:40.789 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:30:40.791 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:30:40.791 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:30:40.791 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:30:40.791 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:30:40.791 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 176 = '' 2024-09-17T23:30:40.791 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:30:40.792 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.8 2024-09-17T23:30:41.593 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_46 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_46.status 2024-09-17T23:30:41.593 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:30:41.593 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:30:41.593 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_46 2024-09-17T23:30:41.594 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_46.status 2024-09-17T23:30:41.594 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_46 --format xml 2024-09-17T23:30:41.594 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:30:41.640 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:30:41.642 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id=176 2024-09-17T23:30:41.643 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 176 = 176 2024-09-17T23:30:41.643 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:30:41.643 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snap_present cluster1 mirror image_46 snap 2024-09-17T23:30:41.643 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:30:41.643 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:30:41.643 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_46 2024-09-17T23:30:41.643 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:30:41.643 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:30:41.643 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-17T23:30:41.643 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:30:42.260 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-17T23:30:42.256+0000 7f0394bcc640 -1 reset not still connected to 0x5605454ba4e0 2024-09-17T23:30:42.644 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 info mirror/image_46@snap 2024-09-17T23:30:42.693 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'image_46': 2024-09-17T23:30:42.693 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 128 MiB in 32 objects 2024-09-17T23:30:42.693 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:30:42.693 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 2 2024-09-17T23:30:42.693 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 12106d54134b 2024-09-17T23:30:42.693 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.12106d54134b 2024-09-17T23:30:42.693 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:30:42.693 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-17T23:30:42.693 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:30:42.693 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:30:42.693 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:24:14 2024 2024-09-17T23:30:42.694 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:24:14 2024 2024-09-17T23:30:42.694 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:24:14 2024 2024-09-17T23:30:42.694 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: protected: False 2024-09-17T23:30:42.694 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:30:42.694 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:30:42.694 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: bb8f076b-e4d3-47ce-9e64-72dd50f71318 2024-09-17T23:30:42.694 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: false 2024-09-17T23:30:42.697 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:30:42.697 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ compare_image_snaps mirror image_46 snap 2024-09-17T23:30:42.697 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:30:42.697 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_46 2024-09-17T23:30:42.697 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:30:42.697 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local ret=0 2024-09-17T23:30:42.697 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local rmt_export=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_46.export 2024-09-17T23:30:42.698 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local loc_export=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_46.export 2024-09-17T23:30:42.698 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_46.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_46.export 2024-09-17T23:30:42.698 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror export image_46@snap /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_46.export 2024-09-17T23:30:42.874 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-17T23:30:42.880 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 -p mirror export image_46@snap /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_46.export 2024-09-17T23:30:43.091 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-17T23:30:43.099 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ cmp /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_46.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_46.export 2024-09-17T23:30:43.162 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_46.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_46.export 2024-09-17T23:30:43.195 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:30:43.196 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:30:43.196 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_47 2024-09-17T23:30:43.196 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_snap cluster2 mirror image_47 snap 2024-09-17T23:30:43.196 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:30:43.196 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:30:43.196 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_47 2024-09-17T23:30:43.196 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:30:43.196 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror snap create image_47@snap --debug-rbd=20 --debug-journaler=20 2024-09-17T23:30:45.052 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_started cluster1 mirror image_47 2024-09-17T23:30:45.052 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:30:45.053 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:30:45.053 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_47 2024-09-17T23:30:45.053 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_state cluster1 mirror image_47 started 2024-09-17T23:30:45.053 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:30:45.053 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:30:45.053 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_47 2024-09-17T23:30:45.053 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=started 2024-09-17T23:30:45.053 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:30:45.053 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:30:45.053 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:30:46.054 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_replay_state cluster1 mirror image_47 started 2024-09-17T23:30:46.054 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:30:46.054 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:30:46.054 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_47 2024-09-17T23:30:46.054 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=started 2024-09-17T23:30:46.054 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_result 2024-09-17T23:30:46.054 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=stopped 2024-09-17T23:30:46.055 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_47 2024-09-17T23:30:46.055 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_instance=cluster1 2024-09-17T23:30:46.055 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:30:46.055 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ grep -i state 2024-09-17T23:30:46.055 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:30:46.056 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance=cluster1 2024-09-17T23:30:46.056 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local loop_instance 2024-09-17T23:30:46.056 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:30:46.056 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sleep 0 2024-09-17T23:30:46.056 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-17T23:30:46.056 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ seq 0 3 2024-09-17T23:30:46.059 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:30:46.059 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_47 2024-09-17T23:30:46.059 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:0 2024-09-17T23:30:46.059 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:30:46.059 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:30:46.059 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-17T23:30:46.059 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:0 2024-09-17T23:30:46.059 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:30:46.059 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:30:46.059 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:30:46.060 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=0 2024-09-17T23:30:46.060 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-17T23:30:46.060 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:30:46.060 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:30:46.060 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=0 2024-09-17T23:30:46.060 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:30:46.060 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-17T23:30:46.060 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:0 2024-09-17T23:30:46.060 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:30:46.060 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:30:46.060 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-17T23:30:46.060 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:0 2024-09-17T23:30:46.060 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:30:46.061 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:30:46.061 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:30:46.061 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:30:46.061 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-17T23:30:46.061 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:30:46.061 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:30:46.061 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=0 2024-09-17T23:30:46.061 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=0 2024-09-17T23:30:46.061 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-17T23:30:46.081 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:30:46.082 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:30:46.082 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:30:46.082 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_47 2024-09-17T23:30:46.165 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2024-09-17T23:30:46.165 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd mirror status : get status for rbd mirror 2024-09-17T23:30:46.165 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:admin_socket: invalid command 2024-09-17T23:30:46.168 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:30:46.168 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_47 2024-09-17T23:30:46.168 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:1 2024-09-17T23:30:46.168 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:30:46.168 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:30:46.168 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-17T23:30:46.168 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:1 2024-09-17T23:30:46.168 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:30:46.169 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:30:46.169 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:30:46.169 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=1 2024-09-17T23:30:46.169 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-17T23:30:46.169 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:30:46.169 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:30:46.169 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=1 2024-09-17T23:30:46.169 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=1 2024-09-17T23:30:46.169 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-17T23:30:46.169 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:1 2024-09-17T23:30:46.169 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:30:46.169 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:30:46.169 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-17T23:30:46.169 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:1 2024-09-17T23:30:46.170 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:30:46.170 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:30:46.170 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:30:46.170 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=1 2024-09-17T23:30:46.170 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-17T23:30:46.170 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:30:46.170 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:30:46.170 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=1 2024-09-17T23:30:46.170 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=1 2024-09-17T23:30:46.170 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-17T23:30:46.187 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-17T23:30:46.188 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-17T23:30:46.188 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-17T23:30:46.188 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_47 2024-09-17T23:30:46.263 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2024-09-17T23:30:46.264 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd mirror status : get status for rbd mirror 2024-09-17T23:30:46.264 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:admin_socket: invalid command 2024-09-17T23:30:46.266 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:30:46.266 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_47 2024-09-17T23:30:46.266 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:2 2024-09-17T23:30:46.266 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:30:46.267 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:30:46.267 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-17T23:30:46.267 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:2 2024-09-17T23:30:46.267 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:30:46.267 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:30:46.267 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:30:46.267 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=2 2024-09-17T23:30:46.267 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-17T23:30:46.267 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:30:46.267 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:30:46.267 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=2 2024-09-17T23:30:46.267 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=2 2024-09-17T23:30:46.267 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-17T23:30:46.268 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:2 2024-09-17T23:30:46.268 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:30:46.268 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:30:46.268 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-17T23:30:46.268 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:2 2024-09-17T23:30:46.268 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:30:46.268 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:30:46.268 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:30:46.268 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=2 2024-09-17T23:30:46.268 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-17T23:30:46.268 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:30:46.268 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:30:46.268 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=2 2024-09-17T23:30:46.268 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=2 2024-09-17T23:30:46.268 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-17T23:30:46.285 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-17T23:30:46.286 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-17T23:30:46.286 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-17T23:30:46.286 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_47 2024-09-17T23:30:46.406 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ return 0 2024-09-17T23:30:46.407 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ status_result=' "state": "Replaying"' 2024-09-17T23:30:46.407 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ echo ' "state": "Replaying"' 2024-09-17T23:30:46.407 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep -i Replaying 2024-09-17T23:30:46.408 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: "state": "Replaying" 2024-09-17T23:30:46.409 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_state=started 2024-09-17T23:30:46.409 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test started = started 2024-09-17T23:30:46.409 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:30:46.409 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_47 2024-09-17T23:30:46.409 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:30:46.409 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:30:46.409 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:30:46.409 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_47 2024-09-17T23:30:46.410 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = journal ']' 2024-09-17T23:30:46.410 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = snapshot ']' 2024-09-17T23:30:46.410 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_47 2024-09-17T23:30:46.410 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:30:46.410 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:30:46.410 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:30:46.410 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_47 2024-09-17T23:30:46.410 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster2-mirror-image_47.status 2024-09-17T23:30:46.410 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster2-mirror-image_47.status 2024-09-17T23:30:46.410 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:30:46.411 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_47.status 2024-09-17T23:30:46.411 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster1-mirror-image_47.status 2024-09-17T23:30:46.411 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster1-mirror-image_47.status 2024-09-17T23:30:46.411 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:30:46.412 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_status_log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_47.status 2024-09-17T23:30:46.412 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ mirror_image_snapshot cluster2 mirror image_47 2024-09-17T23:30:46.412 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:30:46.412 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:30:46.412 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_47 2024-09-17T23:30:46.412 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_47 2024-09-17T23:30:46.962 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.2709008119014523) > weight (0.25) 2024-09-17T23:30:46.962 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.38372535490299586) > weight (0.25) 2024-09-17T23:30:46.962 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2024-09-17T23:30:46.962 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2024-09-17T23:30:46.963 INFO:tasks.rbd_mirror_thrash:waiting for 9 secs before reviving daemons 2024-09-17T23:30:46.964 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi038.stderr:2024-09-17T23:30:46.961+0000 7fa175c7d640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 53000) UID: 1000 2024-09-17T23:30:47.043 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Snapshot ID: 178 2024-09-17T23:30:47.055 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_47 /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_47.status 2024-09-17T23:30:47.055 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:30:47.055 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:30:47.055 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_47 2024-09-17T23:30:47.055 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_47.status 2024-09-17T23:30:47.055 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_47 --format xml 2024-09-17T23:30:47.055 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:30:47.100 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-17T23:30:47.102 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snapshot_id=178 2024-09-17T23:30:47.102 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:30:47.102 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:30:47.103 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.2 2024-09-17T23:30:47.304 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_47 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_47.status 2024-09-17T23:30:47.304 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:30:47.304 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:30:47.304 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_47 2024-09-17T23:30:47.304 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_47.status 2024-09-17T23:30:47.304 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_47 --format xml 2024-09-17T23:30:47.305 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:30:47.353 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:30:47.353 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:30:47.354 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:30:47.354 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:30:47.354 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:30:47.355 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:30:47.355 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 178 = '' 2024-09-17T23:30:47.355 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:30:47.355 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.4 2024-09-17T23:30:47.756 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_47 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_47.status 2024-09-17T23:30:47.756 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:30:47.756 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:30:47.756 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_47 2024-09-17T23:30:47.756 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_47.status 2024-09-17T23:30:47.756 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_47 --format xml 2024-09-17T23:30:47.757 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:30:47.806 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:30:47.806 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:30:47.807 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:30:47.807 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:30:47.807 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:30:47.808 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:30:47.808 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 178 = '' 2024-09-17T23:30:47.808 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:30:47.808 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.8 2024-09-17T23:30:48.609 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_47 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_47.status 2024-09-17T23:30:48.609 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:30:48.609 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:30:48.609 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_47 2024-09-17T23:30:48.609 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_47.status 2024-09-17T23:30:48.610 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_47 --format xml 2024-09-17T23:30:48.610 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:30:48.655 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:30:48.657 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id=178 2024-09-17T23:30:48.657 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 178 = 178 2024-09-17T23:30:48.657 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:30:48.658 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snap_present cluster1 mirror image_47 snap 2024-09-17T23:30:48.658 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:30:48.658 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:30:48.658 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_47 2024-09-17T23:30:48.658 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:30:48.658 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:30:48.658 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-17T23:30:48.658 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:30:49.658 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 info mirror/image_47@snap 2024-09-17T23:30:49.704 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'image_47': 2024-09-17T23:30:49.704 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 128 MiB in 32 objects 2024-09-17T23:30:49.704 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:30:49.705 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 2 2024-09-17T23:30:49.705 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 1210ac5db5bd 2024-09-17T23:30:49.705 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.1210ac5db5bd 2024-09-17T23:30:49.705 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:30:49.705 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-17T23:30:49.705 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:30:49.705 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:30:49.705 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:24:21 2024 2024-09-17T23:30:49.705 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:24:21 2024 2024-09-17T23:30:49.705 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:24:21 2024 2024-09-17T23:30:49.705 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: protected: False 2024-09-17T23:30:49.706 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:30:49.706 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:30:49.706 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: 04e3e2c8-dba3-4aac-a2db-a1e1fd0c90d3 2024-09-17T23:30:49.706 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: false 2024-09-17T23:30:49.708 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:30:49.708 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ compare_image_snaps mirror image_47 snap 2024-09-17T23:30:49.708 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:30:49.708 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_47 2024-09-17T23:30:49.708 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:30:49.708 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local ret=0 2024-09-17T23:30:49.708 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local rmt_export=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_47.export 2024-09-17T23:30:49.708 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local loc_export=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_47.export 2024-09-17T23:30:49.709 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_47.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_47.export 2024-09-17T23:30:49.709 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror export image_47@snap /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_47.export 2024-09-17T23:30:49.877 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-17T23:30:49.884 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 -p mirror export image_47@snap /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_47.export 2024-09-17T23:30:50.099 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-17T23:30:50.111 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ cmp /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_47.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_47.export 2024-09-17T23:30:50.174 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_47.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_47.export 2024-09-17T23:30:50.206 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:30:50.207 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:30:50.207 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_48 2024-09-17T23:30:50.207 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_snap cluster2 mirror image_48 snap 2024-09-17T23:30:50.207 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:30:50.207 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:30:50.207 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_48 2024-09-17T23:30:50.207 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:30:50.207 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror snap create image_48@snap --debug-rbd=20 --debug-journaler=20 2024-09-17T23:30:52.065 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_started cluster1 mirror image_48 2024-09-17T23:30:52.065 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:30:52.065 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:30:52.065 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_48 2024-09-17T23:30:52.065 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_state cluster1 mirror image_48 started 2024-09-17T23:30:52.065 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:30:52.066 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:30:52.066 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_48 2024-09-17T23:30:52.066 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=started 2024-09-17T23:30:52.066 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:30:52.066 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:30:52.066 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:30:53.067 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_replay_state cluster1 mirror image_48 started 2024-09-17T23:30:53.067 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:30:53.067 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:30:53.067 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_48 2024-09-17T23:30:53.067 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=started 2024-09-17T23:30:53.067 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_result 2024-09-17T23:30:53.067 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=stopped 2024-09-17T23:30:53.068 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_48 2024-09-17T23:30:53.069 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_instance=cluster1 2024-09-17T23:30:53.069 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:30:53.069 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:30:53.069 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ grep -i state 2024-09-17T23:30:53.069 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance=cluster1 2024-09-17T23:30:53.069 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local loop_instance 2024-09-17T23:30:53.069 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:30:53.069 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sleep 0 2024-09-17T23:30:53.069 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-17T23:30:53.069 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ seq 0 3 2024-09-17T23:30:53.072 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:30:53.072 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_48 2024-09-17T23:30:53.072 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:0 2024-09-17T23:30:53.072 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:30:53.072 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:30:53.072 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-17T23:30:53.072 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:0 2024-09-17T23:30:53.072 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:30:53.072 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:30:53.073 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:30:53.073 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=0 2024-09-17T23:30:53.073 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-17T23:30:53.073 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:30:53.073 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:30:53.073 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=0 2024-09-17T23:30:53.073 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:30:53.073 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-17T23:30:53.073 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:0 2024-09-17T23:30:53.073 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:30:53.073 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:30:53.073 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-17T23:30:53.073 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:0 2024-09-17T23:30:53.074 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:30:53.074 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:30:53.074 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:30:53.074 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:30:53.074 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-17T23:30:53.074 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:30:53.074 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:30:53.074 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=0 2024-09-17T23:30:53.074 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=0 2024-09-17T23:30:53.074 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-17T23:30:53.093 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:30:53.093 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:30:53.093 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:30:53.093 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_48 2024-09-17T23:30:53.174 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:Invalid command: unused arguments: ['mirror/image_48'] 2024-09-17T23:30:53.175 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd mirror status : get status for rbd mirror 2024-09-17T23:30:53.175 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:admin_socket: invalid command 2024-09-17T23:30:53.178 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:30:53.178 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_48 2024-09-17T23:30:53.178 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:1 2024-09-17T23:30:53.178 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:30:53.178 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:30:53.178 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-17T23:30:53.178 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:1 2024-09-17T23:30:53.178 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:30:53.178 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:30:53.178 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:30:53.178 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=1 2024-09-17T23:30:53.179 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-17T23:30:53.179 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:30:53.179 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:30:53.179 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=1 2024-09-17T23:30:53.179 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=1 2024-09-17T23:30:53.179 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-17T23:30:53.179 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:1 2024-09-17T23:30:53.179 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:30:53.179 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:30:53.179 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-17T23:30:53.179 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:1 2024-09-17T23:30:53.179 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:30:53.179 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:30:53.179 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:30:53.180 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=1 2024-09-17T23:30:53.180 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-17T23:30:53.180 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:30:53.180 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:30:53.180 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=1 2024-09-17T23:30:53.180 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=1 2024-09-17T23:30:53.180 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-17T23:30:53.197 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-17T23:30:53.197 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-17T23:30:53.197 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-17T23:30:53.197 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_48 2024-09-17T23:30:53.273 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:Invalid command: unused arguments: ['mirror/image_48'] 2024-09-17T23:30:53.273 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd mirror status : get status for rbd mirror 2024-09-17T23:30:53.273 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:admin_socket: invalid command 2024-09-17T23:30:53.276 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:30:53.276 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_48 2024-09-17T23:30:53.276 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:2 2024-09-17T23:30:53.276 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:30:53.276 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:30:53.276 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-17T23:30:53.276 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:2 2024-09-17T23:30:53.276 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:30:53.277 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:30:53.277 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:30:53.277 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=2 2024-09-17T23:30:53.277 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-17T23:30:53.277 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:30:53.277 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:30:53.277 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=2 2024-09-17T23:30:53.277 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=2 2024-09-17T23:30:53.277 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-17T23:30:53.277 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:2 2024-09-17T23:30:53.277 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:30:53.277 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:30:53.277 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-17T23:30:53.277 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:2 2024-09-17T23:30:53.278 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:30:53.278 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:30:53.278 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:30:53.278 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=2 2024-09-17T23:30:53.278 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-17T23:30:53.278 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:30:53.278 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:30:53.278 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=2 2024-09-17T23:30:53.278 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=2 2024-09-17T23:30:53.278 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-17T23:30:53.295 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-17T23:30:53.295 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-17T23:30:53.295 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-17T23:30:53.295 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_48 2024-09-17T23:30:53.416 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ return 0 2024-09-17T23:30:53.416 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ status_result=' "state": "Replaying"' 2024-09-17T23:30:53.416 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ echo ' "state": "Replaying"' 2024-09-17T23:30:53.416 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep -i Replaying 2024-09-17T23:30:53.417 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: "state": "Replaying" 2024-09-17T23:30:53.419 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_state=started 2024-09-17T23:30:53.419 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test started = started 2024-09-17T23:30:53.419 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:30:53.419 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_48 2024-09-17T23:30:53.419 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:30:53.419 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:30:53.419 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:30:53.419 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_48 2024-09-17T23:30:53.419 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = journal ']' 2024-09-17T23:30:53.419 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = snapshot ']' 2024-09-17T23:30:53.420 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_48 2024-09-17T23:30:53.420 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:30:53.420 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:30:53.420 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:30:53.420 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_48 2024-09-17T23:30:53.420 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster2-mirror-image_48.status 2024-09-17T23:30:53.420 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster2-mirror-image_48.status 2024-09-17T23:30:53.420 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:30:53.421 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_48.status 2024-09-17T23:30:53.421 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster1-mirror-image_48.status 2024-09-17T23:30:53.421 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster1-mirror-image_48.status 2024-09-17T23:30:53.421 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:30:53.421 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_status_log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_48.status 2024-09-17T23:30:53.422 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ mirror_image_snapshot cluster2 mirror image_48 2024-09-17T23:30:53.422 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:30:53.422 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:30:53.422 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_48 2024-09-17T23:30:53.422 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_48 2024-09-17T23:30:54.075 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Snapshot ID: 180 2024-09-17T23:30:54.088 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_48 /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_48.status 2024-09-17T23:30:54.088 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:30:54.088 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:30:54.088 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_48 2024-09-17T23:30:54.088 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_48.status 2024-09-17T23:30:54.088 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_48 --format xml 2024-09-17T23:30:54.088 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:30:54.136 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-17T23:30:54.138 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snapshot_id=180 2024-09-17T23:30:54.138 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:30:54.138 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:30:54.138 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.2 2024-09-17T23:30:54.339 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_48 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_48.status 2024-09-17T23:30:54.340 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:30:54.340 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:30:54.340 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_48 2024-09-17T23:30:54.340 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_48.status 2024-09-17T23:30:54.340 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_48 --format xml 2024-09-17T23:30:54.340 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:30:54.392 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:30:54.392 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:30:54.394 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:30:54.394 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:30:54.394 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:30:54.394 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:30:54.394 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 180 = '' 2024-09-17T23:30:54.395 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:30:54.395 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.4 2024-09-17T23:30:54.796 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_48 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_48.status 2024-09-17T23:30:54.796 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:30:54.796 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:30:54.796 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_48 2024-09-17T23:30:54.796 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_48.status 2024-09-17T23:30:54.796 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_48 --format xml 2024-09-17T23:30:54.797 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:30:54.849 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:30:54.850 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:30:54.851 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:30:54.851 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:30:54.851 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:30:54.852 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:30:54.852 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 180 = '' 2024-09-17T23:30:54.852 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:30:54.852 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.8 2024-09-17T23:30:55.654 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_48 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_48.status 2024-09-17T23:30:55.654 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:30:55.654 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:30:55.654 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_48 2024-09-17T23:30:55.654 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_48.status 2024-09-17T23:30:55.654 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_48 --format xml 2024-09-17T23:30:55.654 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:30:55.701 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:30:55.703 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id=180 2024-09-17T23:30:55.703 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 180 = 180 2024-09-17T23:30:55.703 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:30:55.703 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snap_present cluster1 mirror image_48 snap 2024-09-17T23:30:55.703 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:30:55.703 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:30:55.703 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_48 2024-09-17T23:30:55.703 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:30:55.703 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:30:55.703 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-17T23:30:55.704 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:30:55.963 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2024-09-17T23:30:55.963 INFO:teuthology.orchestra.run:waiting for 600 2024-09-17T23:30:55.964 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2024-09-17T23:30:55.964 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2024-09-17T23:30:55.964 DEBUG:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2024-09-17T23:30:55.967 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2024-09-17T23:30:55.967 INFO:tasks.rbd_mirror_thrash:waiting for 60 secs before thrashing 2024-09-17T23:30:56.704 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 info mirror/image_48@snap 2024-09-17T23:30:56.749 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'image_48': 2024-09-17T23:30:56.749 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 128 MiB in 32 objects 2024-09-17T23:30:56.750 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:30:56.750 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 2 2024-09-17T23:30:56.750 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 12102ddbf4d5 2024-09-17T23:30:56.750 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.12102ddbf4d5 2024-09-17T23:30:56.750 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:30:56.750 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-17T23:30:56.750 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:30:56.750 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:30:56.750 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:24:27 2024 2024-09-17T23:30:56.750 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:24:27 2024 2024-09-17T23:30:56.750 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:24:27 2024 2024-09-17T23:30:56.750 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: protected: False 2024-09-17T23:30:56.750 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:30:56.750 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:30:56.751 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: 38a2f93b-1926-44fa-95d5-32067464b867 2024-09-17T23:30:56.751 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: false 2024-09-17T23:30:56.753 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:30:56.753 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ compare_image_snaps mirror image_48 snap 2024-09-17T23:30:56.753 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:30:56.753 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_48 2024-09-17T23:30:56.753 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:30:56.753 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local ret=0 2024-09-17T23:30:56.754 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local rmt_export=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_48.export 2024-09-17T23:30:56.754 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local loc_export=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_48.export 2024-09-17T23:30:56.754 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_48.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_48.export 2024-09-17T23:30:56.754 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror export image_48@snap /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_48.export 2024-09-17T23:30:56.936 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-17T23:30:56.948 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 -p mirror export image_48@snap /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_48.export 2024-09-17T23:30:57.198 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-17T23:30:57.206 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ cmp /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_48.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_48.export 2024-09-17T23:30:57.269 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_48.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_48.export 2024-09-17T23:30:57.301 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:30:57.301 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:30:57.301 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_49 2024-09-17T23:30:57.301 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_snap cluster2 mirror image_49 snap 2024-09-17T23:30:57.302 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:30:57.302 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:30:57.302 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_49 2024-09-17T23:30:57.302 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:30:57.302 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror snap create image_49@snap --debug-rbd=20 --debug-journaler=20 2024-09-17T23:30:59.113 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_started cluster1 mirror image_49 2024-09-17T23:30:59.114 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:30:59.114 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:30:59.114 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_49 2024-09-17T23:30:59.114 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_state cluster1 mirror image_49 started 2024-09-17T23:30:59.114 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:30:59.114 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:30:59.114 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_49 2024-09-17T23:30:59.114 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=started 2024-09-17T23:30:59.114 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:30:59.114 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:30:59.114 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:31:00.114 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_replay_state cluster1 mirror image_49 started 2024-09-17T23:31:00.115 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:31:00.115 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:31:00.115 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_49 2024-09-17T23:31:00.115 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=started 2024-09-17T23:31:00.115 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_result 2024-09-17T23:31:00.115 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=stopped 2024-09-17T23:31:00.116 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_49 2024-09-17T23:31:00.116 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_instance=cluster1 2024-09-17T23:31:00.116 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:31:00.117 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ grep -i state 2024-09-17T23:31:00.117 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:31:00.117 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance=cluster1 2024-09-17T23:31:00.117 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local loop_instance 2024-09-17T23:31:00.117 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:31:00.117 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sleep 0 2024-09-17T23:31:00.117 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-17T23:31:00.120 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ seq 0 3 2024-09-17T23:31:00.120 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:31:00.120 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_49 2024-09-17T23:31:00.120 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:0 2024-09-17T23:31:00.120 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:31:00.120 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:31:00.120 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-17T23:31:00.120 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:0 2024-09-17T23:31:00.121 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:31:00.121 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:31:00.121 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:31:00.121 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=0 2024-09-17T23:31:00.121 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-17T23:31:00.121 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:31:00.121 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:31:00.121 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=0 2024-09-17T23:31:00.121 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:31:00.121 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-17T23:31:00.121 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:0 2024-09-17T23:31:00.121 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:31:00.121 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:31:00.121 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-17T23:31:00.122 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:0 2024-09-17T23:31:00.122 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:31:00.122 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:31:00.122 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:31:00.122 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:31:00.122 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-17T23:31:00.122 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:31:00.122 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:31:00.122 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=0 2024-09-17T23:31:00.122 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=0 2024-09-17T23:31:00.122 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-17T23:31:00.142 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:31:00.143 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:31:00.143 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:31:00.143 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_49 2024-09-17T23:31:00.226 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:Invalid command: unused arguments: ['mirror/image_49'] 2024-09-17T23:31:00.226 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd mirror status : get status for rbd mirror 2024-09-17T23:31:00.226 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:admin_socket: invalid command 2024-09-17T23:31:00.229 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:31:00.229 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_49 2024-09-17T23:31:00.229 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:1 2024-09-17T23:31:00.229 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:31:00.229 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:31:00.230 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-17T23:31:00.230 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:1 2024-09-17T23:31:00.230 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:31:00.230 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:31:00.230 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:31:00.230 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=1 2024-09-17T23:31:00.230 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-17T23:31:00.230 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:31:00.230 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:31:00.230 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=1 2024-09-17T23:31:00.230 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=1 2024-09-17T23:31:00.230 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-17T23:31:00.230 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:1 2024-09-17T23:31:00.231 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:31:00.231 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:31:00.231 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-17T23:31:00.231 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:1 2024-09-17T23:31:00.231 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:31:00.231 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:31:00.231 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:31:00.231 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=1 2024-09-17T23:31:00.231 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-17T23:31:00.231 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:31:00.231 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:31:00.231 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=1 2024-09-17T23:31:00.232 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=1 2024-09-17T23:31:00.232 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-17T23:31:00.249 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-17T23:31:00.250 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-17T23:31:00.250 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-17T23:31:00.250 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_49 2024-09-17T23:31:00.327 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:Invalid command: unused arguments: ['mirror/image_49'] 2024-09-17T23:31:00.327 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd mirror status : get status for rbd mirror 2024-09-17T23:31:00.327 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:admin_socket: invalid command 2024-09-17T23:31:00.329 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:31:00.330 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_49 2024-09-17T23:31:00.330 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:2 2024-09-17T23:31:00.330 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:31:00.330 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:31:00.330 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-17T23:31:00.330 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:2 2024-09-17T23:31:00.330 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:31:00.330 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:31:00.330 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:31:00.330 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=2 2024-09-17T23:31:00.330 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-17T23:31:00.330 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:31:00.330 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:31:00.330 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=2 2024-09-17T23:31:00.331 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=2 2024-09-17T23:31:00.331 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-17T23:31:00.331 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:2 2024-09-17T23:31:00.331 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:31:00.331 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:31:00.331 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-17T23:31:00.331 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:2 2024-09-17T23:31:00.331 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:31:00.331 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:31:00.331 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:31:00.331 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=2 2024-09-17T23:31:00.331 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-17T23:31:00.331 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:31:00.331 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:31:00.331 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=2 2024-09-17T23:31:00.332 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=2 2024-09-17T23:31:00.332 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-17T23:31:00.348 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-17T23:31:00.349 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-17T23:31:00.349 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-17T23:31:00.349 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_49 2024-09-17T23:31:00.469 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ return 0 2024-09-17T23:31:00.469 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ status_result=' "state": "Replaying"' 2024-09-17T23:31:00.469 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ echo ' "state": "Replaying"' 2024-09-17T23:31:00.469 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep -i Replaying 2024-09-17T23:31:00.470 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: "state": "Replaying" 2024-09-17T23:31:00.471 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_state=started 2024-09-17T23:31:00.471 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test started = started 2024-09-17T23:31:00.471 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:31:00.472 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_49 2024-09-17T23:31:00.472 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:31:00.472 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:31:00.472 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:31:00.472 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_49 2024-09-17T23:31:00.472 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = journal ']' 2024-09-17T23:31:00.472 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = snapshot ']' 2024-09-17T23:31:00.472 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_49 2024-09-17T23:31:00.472 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:31:00.472 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:31:00.472 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:31:00.472 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_49 2024-09-17T23:31:00.472 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster2-mirror-image_49.status 2024-09-17T23:31:00.472 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster2-mirror-image_49.status 2024-09-17T23:31:00.473 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:31:00.474 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_49.status 2024-09-17T23:31:00.474 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster1-mirror-image_49.status 2024-09-17T23:31:00.474 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster1-mirror-image_49.status 2024-09-17T23:31:00.475 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:31:00.475 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_status_log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_49.status 2024-09-17T23:31:00.475 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ mirror_image_snapshot cluster2 mirror image_49 2024-09-17T23:31:00.475 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:31:00.475 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:31:00.475 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_49 2024-09-17T23:31:00.475 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_49 2024-09-17T23:31:01.124 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Snapshot ID: 182 2024-09-17T23:31:01.137 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_49 /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_49.status 2024-09-17T23:31:01.138 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:31:01.138 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:31:01.138 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_49 2024-09-17T23:31:01.138 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_49.status 2024-09-17T23:31:01.138 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_49 --format xml 2024-09-17T23:31:01.138 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:31:01.184 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-17T23:31:01.186 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snapshot_id=182 2024-09-17T23:31:01.186 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:31:01.186 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:31:01.186 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.2 2024-09-17T23:31:01.387 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_49 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_49.status 2024-09-17T23:31:01.387 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:31:01.388 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:31:01.388 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_49 2024-09-17T23:31:01.388 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_49.status 2024-09-17T23:31:01.388 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_49 --format xml 2024-09-17T23:31:01.388 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:31:01.439 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:31:01.440 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:31:01.441 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:31:01.441 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:31:01.441 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:31:01.442 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:31:01.442 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 182 = '' 2024-09-17T23:31:01.442 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:31:01.442 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.4 2024-09-17T23:31:01.844 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_49 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_49.status 2024-09-17T23:31:01.844 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:31:01.844 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:31:01.844 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_49 2024-09-17T23:31:01.844 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_49.status 2024-09-17T23:31:01.845 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_49 --format xml 2024-09-17T23:31:01.845 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:31:01.897 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:31:01.897 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:31:01.899 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:31:01.899 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:31:01.899 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:31:01.899 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:31:01.899 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 182 = '' 2024-09-17T23:31:01.900 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:31:01.900 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.8 2024-09-17T23:31:02.701 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_49 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_49.status 2024-09-17T23:31:02.701 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:31:02.701 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:31:02.701 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_49 2024-09-17T23:31:02.701 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_49.status 2024-09-17T23:31:02.701 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_49 --format xml 2024-09-17T23:31:02.701 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:31:02.749 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:31:02.753 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id=182 2024-09-17T23:31:02.753 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 182 = 182 2024-09-17T23:31:02.754 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:31:02.754 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snap_present cluster1 mirror image_49 snap 2024-09-17T23:31:02.754 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:31:02.754 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:31:02.754 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_49 2024-09-17T23:31:02.754 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:31:02.754 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:31:02.754 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-17T23:31:02.754 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:31:03.752 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 info mirror/image_49@snap 2024-09-17T23:31:03.802 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'image_49': 2024-09-17T23:31:03.802 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 128 MiB in 32 objects 2024-09-17T23:31:03.802 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:31:03.802 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 2 2024-09-17T23:31:03.802 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 1210fb636f37 2024-09-17T23:31:03.802 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.1210fb636f37 2024-09-17T23:31:03.802 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:31:03.802 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-17T23:31:03.802 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:31:03.802 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:31:03.802 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:24:41 2024 2024-09-17T23:31:03.803 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:24:41 2024 2024-09-17T23:31:03.803 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:24:41 2024 2024-09-17T23:31:03.803 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: protected: False 2024-09-17T23:31:03.803 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:31:03.803 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:31:03.803 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: 61b5b6dd-6f04-4689-abba-edcd8570ecec 2024-09-17T23:31:03.803 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: false 2024-09-17T23:31:03.806 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:31:03.806 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ compare_image_snaps mirror image_49 snap 2024-09-17T23:31:03.806 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:31:03.806 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_49 2024-09-17T23:31:03.806 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:31:03.806 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local ret=0 2024-09-17T23:31:03.806 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local rmt_export=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_49.export 2024-09-17T23:31:03.806 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local loc_export=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_49.export 2024-09-17T23:31:03.806 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_49.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_49.export 2024-09-17T23:31:03.807 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror export image_49@snap /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_49.export 2024-09-17T23:31:03.985 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-17T23:31:03.992 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 -p mirror export image_49@snap /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_49.export 2024-09-17T23:31:04.235 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-17T23:31:04.243 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ cmp /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_49.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_49.export 2024-09-17T23:31:04.306 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_49.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_49.export 2024-09-17T23:31:04.339 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:31:04.339 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:31:04.339 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_50 2024-09-17T23:31:04.339 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_snap cluster2 mirror image_50 snap 2024-09-17T23:31:04.339 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:31:04.339 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:31:04.340 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_50 2024-09-17T23:31:04.340 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:31:04.340 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror snap create image_50@snap --debug-rbd=20 --debug-journaler=20 2024-09-17T23:31:06.137 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_started cluster1 mirror image_50 2024-09-17T23:31:06.137 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:31:06.138 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:31:06.138 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_50 2024-09-17T23:31:06.138 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_replay_state cluster1 mirror image_50 started 2024-09-17T23:31:06.138 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:31:06.138 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:31:06.138 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_50 2024-09-17T23:31:06.138 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=started 2024-09-17T23:31:06.138 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:31:06.138 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:31:06.138 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:31:07.139 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_replay_state cluster1 mirror image_50 started 2024-09-17T23:31:07.139 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:31:07.139 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:31:07.139 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_50 2024-09-17T23:31:07.139 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=started 2024-09-17T23:31:07.139 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_result 2024-09-17T23:31:07.139 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=stopped 2024-09-17T23:31:07.140 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_50 2024-09-17T23:31:07.140 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_instance=cluster1 2024-09-17T23:31:07.140 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:31:07.140 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ grep -i state 2024-09-17T23:31:07.141 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:31:07.141 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance=cluster1 2024-09-17T23:31:07.141 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local loop_instance 2024-09-17T23:31:07.141 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:31:07.141 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sleep 0 2024-09-17T23:31:07.141 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-17T23:31:07.142 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ seq 0 3 2024-09-17T23:31:07.142 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:31:07.142 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_50 2024-09-17T23:31:07.143 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:0 2024-09-17T23:31:07.143 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:31:07.143 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:31:07.143 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-17T23:31:07.143 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:0 2024-09-17T23:31:07.143 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:31:07.143 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:31:07.143 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:31:07.143 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=0 2024-09-17T23:31:07.143 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-17T23:31:07.143 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:31:07.143 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:31:07.145 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=0 2024-09-17T23:31:07.145 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:31:07.145 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-17T23:31:07.145 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:0 2024-09-17T23:31:07.145 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:31:07.146 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:31:07.146 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-17T23:31:07.146 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:0 2024-09-17T23:31:07.146 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:31:07.146 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:31:07.146 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:31:07.146 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=0 2024-09-17T23:31:07.146 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-17T23:31:07.146 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:31:07.146 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:31:07.146 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=0 2024-09-17T23:31:07.146 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=0 2024-09-17T23:31:07.146 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-17T23:31:07.166 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:31:07.167 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:31:07.167 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-17T23:31:07.167 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_50 2024-09-17T23:31:07.248 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:Invalid command: unused arguments: ['mirror/image_50'] 2024-09-17T23:31:07.248 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd mirror status : get status for rbd mirror 2024-09-17T23:31:07.249 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:admin_socket: invalid command 2024-09-17T23:31:07.254 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:31:07.254 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_50 2024-09-17T23:31:07.254 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:1 2024-09-17T23:31:07.254 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:31:07.254 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:31:07.254 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-17T23:31:07.254 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:1 2024-09-17T23:31:07.254 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:31:07.254 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:31:07.254 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:31:07.254 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=1 2024-09-17T23:31:07.255 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-17T23:31:07.255 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:31:07.255 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:31:07.255 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=1 2024-09-17T23:31:07.255 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=1 2024-09-17T23:31:07.255 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-17T23:31:07.255 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:1 2024-09-17T23:31:07.255 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:31:07.255 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:31:07.255 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-17T23:31:07.255 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:1 2024-09-17T23:31:07.255 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:31:07.255 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:31:07.256 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:31:07.256 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=1 2024-09-17T23:31:07.256 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-17T23:31:07.256 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:31:07.256 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:31:07.256 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=1 2024-09-17T23:31:07.256 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=1 2024-09-17T23:31:07.256 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-17T23:31:07.271 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-17T23:31:07.271 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-17T23:31:07.271 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-17T23:31:07.271 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_50 2024-09-17T23:31:07.347 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:Invalid command: unused arguments: ['mirror/image_50'] 2024-09-17T23:31:07.347 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd mirror status : get status for rbd mirror 2024-09-17T23:31:07.347 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:admin_socket: invalid command 2024-09-17T23:31:07.350 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:31:07.350 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_50 2024-09-17T23:31:07.350 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1:2 2024-09-17T23:31:07.350 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ shift 2024-09-17T23:31:07.350 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance 2024-09-17T23:31:07.350 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-17T23:31:07.350 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local val=cluster1:2 2024-09-17T23:31:07.351 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster_var_name=cluster 2024-09-17T23:31:07.351 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local instance_var_name=instance 2024-09-17T23:31:07.351 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:31:07.351 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=2 2024-09-17T23:31:07.351 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-17T23:31:07.351 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval cluster=cluster1 2024-09-17T23:31:07.351 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:31:07.351 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ eval instance=2 2024-09-17T23:31:07.351 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=2 2024-09-17T23:31:07.351 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-17T23:31:07.351 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local local_cluster=cluster1:2 2024-09-17T23:31:07.351 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster=cluster1 2024-09-17T23:31:07.351 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance 2024-09-17T23:31:07.351 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-17T23:31:07.352 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local val=cluster1:2 2024-09-17T23:31:07.352 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local cluster_var_name=local_cluster 2024-09-17T23:31:07.352 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ local instance_var_name=instance 2024-09-17T23:31:07.352 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ cluster=cluster1 2024-09-17T23:31:07.352 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ instance=2 2024-09-17T23:31:07.352 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-17T23:31:07.352 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval local_cluster=cluster1 2024-09-17T23:31:07.352 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ local_cluster=cluster1 2024-09-17T23:31:07.352 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ eval instance=2 2024-09-17T23:31:07.352 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ instance=2 2024-09-17T23:31:07.352 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-17T23:31:07.369 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-17T23:31:07.369 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-17T23:31:07.369 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-17T23:31:07.369 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_50 2024-09-17T23:31:07.488 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ return 0 2024-09-17T23:31:07.489 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ status_result=' "state": "Replaying"' 2024-09-17T23:31:07.489 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ echo ' "state": "Replaying"' 2024-09-17T23:31:07.489 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ grep -i Replaying 2024-09-17T23:31:07.490 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: "state": "Replaying" 2024-09-17T23:31:07.491 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_state=started 2024-09-17T23:31:07.491 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test started = started 2024-09-17T23:31:07.491 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:31:07.491 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_50 2024-09-17T23:31:07.492 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:31:07.492 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:31:07.492 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:31:07.492 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_50 2024-09-17T23:31:07.492 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = journal ']' 2024-09-17T23:31:07.492 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' snapshot = snapshot ']' 2024-09-17T23:31:07.492 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_50 2024-09-17T23:31:07.492 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_cluster=cluster1 2024-09-17T23:31:07.492 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:31:07.492 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:31:07.492 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_50 2024-09-17T23:31:07.492 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster2-mirror-image_50.status 2024-09-17T23:31:07.492 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster2-mirror-image_50.status 2024-09-17T23:31:07.493 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:31:07.493 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local status_log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_50.status 2024-09-17T23:31:07.493 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ mkfname cluster1-mirror-image_50.status 2024-09-17T23:31:07.493 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ echo cluster1-mirror-image_50.status 2024-09-17T23:31:07.493 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-17T23:31:07.494 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local local_status_log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_50.status 2024-09-17T23:31:07.494 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ mirror_image_snapshot cluster2 mirror image_50 2024-09-17T23:31:07.494 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:31:07.494 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:31:07.495 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_50 2024-09-17T23:31:07.495 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_50 2024-09-17T23:31:08.146 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Snapshot ID: 184 2024-09-17T23:31:08.160 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_50 /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_50.status 2024-09-17T23:31:08.160 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:31:08.160 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:31:08.160 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_50 2024-09-17T23:31:08.161 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_50.status 2024-09-17T23:31:08.161 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_50 --format xml 2024-09-17T23:31:08.161 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:31:08.209 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-17T23:31:08.211 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snapshot_id=184 2024-09-17T23:31:08.211 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:31:08.211 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:31:08.211 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.2 2024-09-17T23:31:08.412 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_50 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_50.status 2024-09-17T23:31:08.412 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:31:08.412 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:31:08.413 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_50 2024-09-17T23:31:08.413 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_50.status 2024-09-17T23:31:08.413 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_50 --format xml 2024-09-17T23:31:08.413 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:31:08.462 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:31:08.462 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:31:08.463 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:31:08.463 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:31:08.463 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:31:08.464 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:31:08.464 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 184 = '' 2024-09-17T23:31:08.464 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:31:08.464 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.4 2024-09-17T23:31:08.865 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_50 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_50.status 2024-09-17T23:31:08.866 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:31:08.866 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:31:08.866 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_50 2024-09-17T23:31:08.866 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_50.status 2024-09-17T23:31:08.866 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_50 --format xml 2024-09-17T23:31:08.866 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:31:09.119 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:31:09.120 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:31:09.121 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:-:1.1: Document is empty 2024-09-17T23:31:09.121 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: 2024-09-17T23:31:09.121 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:^ 2024-09-17T23:31:09.122 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id= 2024-09-17T23:31:09.122 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 184 = '' 2024-09-17T23:31:09.122 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-17T23:31:09.122 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.8 2024-09-17T23:31:09.923 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_50 /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_50.status 2024-09-17T23:31:09.924 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:31:09.924 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:31:09.924 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_50 2024-09-17T23:31:09.924 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local log=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_50.status 2024-09-17T23:31:09.924 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_50 --format xml 2024-09-17T23:31:09.924 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-17T23:31:09.972 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-17T23:31:09.974 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local primary_snapshot_id=184 2024-09-17T23:31:09.974 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 184 = 184 2024-09-17T23:31:09.974 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:31:09.974 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_snap_present cluster1 mirror image_50 snap 2024-09-17T23:31:09.974 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:31:09.974 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:31:09.974 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_50 2024-09-17T23:31:09.974 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:31:09.974 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:31:09.975 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-17T23:31:09.975 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:31:10.975 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 info mirror/image_50@snap 2024-09-17T23:31:11.024 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'image_50': 2024-09-17T23:31:11.024 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 128 MiB in 32 objects 2024-09-17T23:31:11.024 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:31:11.024 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 2 2024-09-17T23:31:11.024 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 1210265fb11f 2024-09-17T23:31:11.024 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.1210265fb11f 2024-09-17T23:31:11.024 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:31:11.024 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-17T23:31:11.025 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:31:11.025 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:31:11.025 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:24:51 2024 2024-09-17T23:31:11.025 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:24:51 2024 2024-09-17T23:31:11.025 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:24:51 2024 2024-09-17T23:31:11.025 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: protected: False 2024-09-17T23:31:11.025 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:31:11.025 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:31:11.025 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: 865bd16a-a0e6-4a3a-bb53-8569e6a216ea 2024-09-17T23:31:11.025 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: false 2024-09-17T23:31:11.028 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:31:11.028 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ compare_image_snaps mirror image_50 snap 2024-09-17T23:31:11.028 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:31:11.028 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_50 2024-09-17T23:31:11.028 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap_name=snap 2024-09-17T23:31:11.028 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local ret=0 2024-09-17T23:31:11.028 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local rmt_export=/tmp/tmp.s5MEFDgsej/cluster2-mirror-image_50.export 2024-09-17T23:31:11.028 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local loc_export=/tmp/tmp.s5MEFDgsej/cluster1-mirror-image_50.export 2024-09-17T23:31:11.029 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_50.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_50.export 2024-09-17T23:31:11.029 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 -p mirror export image_50@snap /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_50.export 2024-09-17T23:31:11.207 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-17T23:31:11.214 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster1 -p mirror export image_50@snap /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_50.export 2024-09-17T23:31:11.437 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-17T23:31:11.444 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ cmp /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_50.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_50.export 2024-09-17T23:31:11.507 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -f /tmp/tmp.s5MEFDgsej/cluster2-mirror-image_50.export /tmp/tmp.s5MEFDgsej/cluster1-mirror-image_50.export 2024-09-17T23:31:11.540 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:31:11.540 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ testlog 'TEST: delete many images' 2024-09-17T23:31:11.540 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ tee -a /tmp/tmp.s5MEFDgsej/rbd-mirror.test.log 2024-09-17T23:31:11.540 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ date '+%F %T' 2024-09-17T23:31:11.548 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ echo 2024-09-17 23:31:11 TEST: delete many images 2024-09-17T23:31:11.548 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:2024-09-17 23:31:11 TEST: delete many images 2024-09-17T23:31:11.548 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ seq 1 50 2024-09-17T23:31:11.549 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:31:11.549 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_1 2024-09-17T23:31:11.549 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_snapshot cluster2 mirror image_1 snap 2024-09-17T23:31:11.550 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:31:11.550 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:31:11.550 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_1 2024-09-17T23:31:11.550 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap=snap 2024-09-17T23:31:11.550 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap rm mirror/image_1@snap 2024-09-17T23:31:12.140 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing snap: 100% complete...done. 2024-09-17T23:31:12.173 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image_retry cluster2 mirror image_1 2024-09-17T23:31:12.174 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:31:12.174 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:31:12.174 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_1 2024-09-17T23:31:12.174 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-17T23:31:12.174 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0 2024-09-17T23:31:12.175 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image cluster2 mirror image_1 2024-09-17T23:31:12.175 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:31:12.175 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:31:12.175 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_1 2024-09-17T23:31:12.175 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_1 2024-09-17T23:31:12.223 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 rm mirror/image_1 2024-09-17T23:31:12.339 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:31:12.337+0000 7ff50242c640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55d142f31520 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-17T23:31:12.339 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:31:12.337+0000 7ff508438640 -1 rbd::mirror::ImageReplayer: 0x55d14318a280 [3/5ca82a92-11a6-461d-96fa-d42ff94d4c65] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-17T23:31:14.283 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-17T23:31:14.286 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:31:14.286 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:31:14.287 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_2 2024-09-17T23:31:14.287 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_snapshot cluster2 mirror image_2 snap 2024-09-17T23:31:14.287 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:31:14.287 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:31:14.287 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_2 2024-09-17T23:31:14.287 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap=snap 2024-09-17T23:31:14.287 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap rm mirror/image_2@snap 2024-09-17T23:31:15.163 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing snap: 100% complete...done. 2024-09-17T23:31:15.185 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image_retry cluster2 mirror image_2 2024-09-17T23:31:15.185 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:31:15.185 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:31:15.186 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_2 2024-09-17T23:31:15.186 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-17T23:31:15.186 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0 2024-09-17T23:31:15.186 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image cluster2 mirror image_2 2024-09-17T23:31:15.187 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:31:15.187 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:31:15.187 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_2 2024-09-17T23:31:15.187 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_2 2024-09-17T23:31:15.448 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 rm mirror/image_2 2024-09-17T23:31:15.575 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:31:15.573+0000 7ff50242c640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55d142f3dd40 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-17T23:31:15.575 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:31:15.573+0000 7ff508438640 -1 rbd::mirror::ImageReplayer: 0x55d14318ac80 [3/e4119d6f-e55a-46f9-97e1-502365b9e208] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-17T23:31:17.287 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-17T23:31:17.293 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:31:17.293 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:31:17.293 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_3 2024-09-17T23:31:17.293 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_snapshot cluster2 mirror image_3 snap 2024-09-17T23:31:17.293 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:31:17.293 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:31:17.293 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_3 2024-09-17T23:31:17.293 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap=snap 2024-09-17T23:31:17.294 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap rm mirror/image_3@snap 2024-09-17T23:31:18.172 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing snap: 100% complete...done. 2024-09-17T23:31:18.203 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image_retry cluster2 mirror image_3 2024-09-17T23:31:18.203 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:31:18.203 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:31:18.203 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_3 2024-09-17T23:31:18.203 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-17T23:31:18.203 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0 2024-09-17T23:31:18.204 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image cluster2 mirror image_3 2024-09-17T23:31:18.204 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:31:18.204 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:31:18.204 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_3 2024-09-17T23:31:18.204 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_3 2024-09-17T23:31:18.253 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 rm mirror/image_3 2024-09-17T23:31:18.573 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:31:18.571+0000 7ff501c2b640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55d142f3e000 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-17T23:31:18.573 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:31:18.571+0000 7ff508438640 -1 rbd::mirror::ImageReplayer: 0x55d14318af00 [3/cb923d84-7e5a-4d91-9543-3952a9f3edb7] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-17T23:31:20.320 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-17T23:31:20.321 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:31:20.321 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:31:20.321 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_4 2024-09-17T23:31:20.321 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_snapshot cluster2 mirror image_4 snap 2024-09-17T23:31:20.322 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:31:20.322 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:31:20.322 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_4 2024-09-17T23:31:20.322 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap=snap 2024-09-17T23:31:20.322 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap rm mirror/image_4@snap 2024-09-17T23:31:21.201 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing snap: 100% complete...done. 2024-09-17T23:31:21.230 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image_retry cluster2 mirror image_4 2024-09-17T23:31:21.231 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:31:21.231 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:31:21.231 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_4 2024-09-17T23:31:21.231 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-17T23:31:21.231 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0 2024-09-17T23:31:21.231 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image cluster2 mirror image_4 2024-09-17T23:31:21.231 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:31:21.231 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:31:21.232 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_4 2024-09-17T23:31:21.232 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_4 2024-09-17T23:31:21.282 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 rm mirror/image_4 2024-09-17T23:31:21.397 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:31:21.393+0000 7ff50242c640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55d142f431e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-17T23:31:21.397 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:31:21.393+0000 7ff508c39640 -1 rbd::mirror::ImageReplayer: 0x55d14318b180 [3/1445a68d-ae17-48f5-9088-ad3e0f9e8951] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-17T23:31:23.328 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-17T23:31:23.332 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:31:23.332 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:31:23.332 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_5 2024-09-17T23:31:23.332 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_snapshot cluster2 mirror image_5 snap 2024-09-17T23:31:23.332 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:31:23.332 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:31:23.333 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_5 2024-09-17T23:31:23.333 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap=snap 2024-09-17T23:31:23.333 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap rm mirror/image_5@snap 2024-09-17T23:31:23.856 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.5196589198447531) > weight (0.25) 2024-09-17T23:31:23.857 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2024-09-17T23:31:23.857 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2024-09-17T23:31:23.857 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2024-09-17T23:31:23.858 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi038.stderr:2024-09-17T23:31:23.856+0000 7fb228a76640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 48642) UID: 1000 2024-09-17T23:31:24.206 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing snap: 100% complete...done. 2024-09-17T23:31:24.234 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image_retry cluster2 mirror image_5 2024-09-17T23:31:24.234 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:31:24.234 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:31:24.234 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_5 2024-09-17T23:31:24.234 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-17T23:31:24.234 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0 2024-09-17T23:31:24.235 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image cluster2 mirror image_5 2024-09-17T23:31:24.235 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:31:24.235 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:31:24.236 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_5 2024-09-17T23:31:24.236 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_5 2024-09-17T23:31:24.288 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 rm mirror/image_5 2024-09-17T23:31:24.403 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:31:24.401+0000 7ff501c2b640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55d142f3e1a0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-17T23:31:24.404 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:31:24.401+0000 7ff508c39640 -1 rbd::mirror::ImageReplayer: 0x55d14318b400 [3/bde8fdf5-e3d2-42d5-b63a-37026e062eaa] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-17T23:31:26.344 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-17T23:31:26.348 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:31:26.348 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:31:26.348 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_6 2024-09-17T23:31:26.348 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_snapshot cluster2 mirror image_6 snap 2024-09-17T23:31:26.348 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:31:26.348 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:31:26.348 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_6 2024-09-17T23:31:26.348 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap=snap 2024-09-17T23:31:26.348 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap rm mirror/image_6@snap 2024-09-17T23:31:27.177 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing snap: 100% complete...done. 2024-09-17T23:31:27.196 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image_retry cluster2 mirror image_6 2024-09-17T23:31:27.196 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:31:27.196 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:31:27.196 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_6 2024-09-17T23:31:27.196 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-17T23:31:27.196 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0 2024-09-17T23:31:27.197 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image cluster2 mirror image_6 2024-09-17T23:31:27.197 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:31:27.197 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:31:27.197 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_6 2024-09-17T23:31:27.197 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_6 2024-09-17T23:31:27.248 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 rm mirror/image_6 2024-09-17T23:31:27.362 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:31:27.360+0000 7ff50242c640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55d14275da00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-17T23:31:27.363 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:31:27.360+0000 7ff508438640 -1 rbd::mirror::ImageReplayer: 0x55d14318b680 [3/d1b13a10-bcbb-4eb9-8b9a-30b9235cece2] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-17T23:31:29.369 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-17T23:31:29.372 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:31:29.373 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:31:29.373 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_7 2024-09-17T23:31:29.373 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_snapshot cluster2 mirror image_7 snap 2024-09-17T23:31:29.373 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:31:29.373 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:31:29.373 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_7 2024-09-17T23:31:29.373 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap=snap 2024-09-17T23:31:29.373 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap rm mirror/image_7@snap 2024-09-17T23:31:30.243 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing snap: 100% complete...done. 2024-09-17T23:31:30.267 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image_retry cluster2 mirror image_7 2024-09-17T23:31:30.267 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:31:30.267 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:31:30.267 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_7 2024-09-17T23:31:30.267 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-17T23:31:30.267 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0 2024-09-17T23:31:30.268 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image cluster2 mirror image_7 2024-09-17T23:31:30.268 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:31:30.268 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:31:30.268 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_7 2024-09-17T23:31:30.268 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_7 2024-09-17T23:31:30.317 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 rm mirror/image_7 2024-09-17T23:31:30.440 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:31:30.437+0000 7ff501c2b640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55d142f31d40 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-17T23:31:30.440 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:31:30.437+0000 7ff508c39640 -1 rbd::mirror::ImageReplayer: 0x55d14318bb80 [3/3eb02a9d-e7e5-427f-9aed-20cdaea8a88e] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-17T23:31:31.858 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2024-09-17T23:31:31.858 INFO:teuthology.orchestra.run:waiting for 600 2024-09-17T23:31:31.859 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2024-09-17T23:31:31.859 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2024-09-17T23:31:31.859 DEBUG:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2024-09-17T23:31:31.862 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2024-09-17T23:31:31.862 INFO:tasks.rbd_mirror_thrash:waiting for 81 secs before thrashing 2024-09-17T23:31:32.281 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-17T23:31:32.284 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:31:32.284 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:31:32.285 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_8 2024-09-17T23:31:32.285 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_snapshot cluster2 mirror image_8 snap 2024-09-17T23:31:32.285 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:31:32.285 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:31:32.285 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_8 2024-09-17T23:31:32.285 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap=snap 2024-09-17T23:31:32.285 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap rm mirror/image_8@snap 2024-09-17T23:31:33.251 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing snap: 100% complete...done. 2024-09-17T23:31:33.281 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image_retry cluster2 mirror image_8 2024-09-17T23:31:33.282 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:31:33.282 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:31:33.282 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_8 2024-09-17T23:31:33.282 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-17T23:31:33.282 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0 2024-09-17T23:31:33.282 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image cluster2 mirror image_8 2024-09-17T23:31:33.283 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:31:33.283 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:31:33.283 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_8 2024-09-17T23:31:33.283 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_8 2024-09-17T23:31:33.335 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 rm mirror/image_8 2024-09-17T23:31:33.452 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:31:33.450+0000 7ff501c2b640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55d142f43380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-17T23:31:33.453 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:31:33.450+0000 7ff508438640 -1 rbd::mirror::ImageReplayer: 0x55d14318b900 [3/ffca36ca-8ea2-4daa-91fd-451fc0c48309] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-17T23:31:35.394 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-17T23:31:35.398 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:31:35.398 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:31:35.398 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_9 2024-09-17T23:31:35.398 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_snapshot cluster2 mirror image_9 snap 2024-09-17T23:31:35.398 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:31:35.398 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:31:35.399 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_9 2024-09-17T23:31:35.399 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap=snap 2024-09-17T23:31:35.399 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap rm mirror/image_9@snap 2024-09-17T23:31:36.264 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing snap: 100% complete...done. 2024-09-17T23:31:36.290 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image_retry cluster2 mirror image_9 2024-09-17T23:31:36.290 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:31:36.290 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:31:36.290 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_9 2024-09-17T23:31:36.290 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-17T23:31:36.290 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0 2024-09-17T23:31:36.291 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image cluster2 mirror image_9 2024-09-17T23:31:36.291 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:31:36.291 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:31:36.292 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_9 2024-09-17T23:31:36.292 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_9 2024-09-17T23:31:36.344 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 rm mirror/image_9 2024-09-17T23:31:36.459 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:31:36.457+0000 7ff501c2b640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55d142f32d00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-17T23:31:36.459 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:31:36.457+0000 7ff508438640 -1 rbd::mirror::ImageReplayer: 0x55d14358a000 [3/bdde2c0b-4959-43dc-9438-c5f0c88af81b] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-17T23:31:38.298 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-17T23:31:38.302 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:31:38.302 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:31:38.302 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_10 2024-09-17T23:31:38.302 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_snapshot cluster2 mirror image_10 snap 2024-09-17T23:31:38.302 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:31:38.302 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:31:38.302 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_10 2024-09-17T23:31:38.303 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap=snap 2024-09-17T23:31:38.303 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap rm mirror/image_10@snap 2024-09-17T23:31:39.294 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing snap: 100% complete...done. 2024-09-17T23:31:39.318 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image_retry cluster2 mirror image_10 2024-09-17T23:31:39.318 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:31:39.318 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:31:39.319 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_10 2024-09-17T23:31:39.319 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-17T23:31:39.319 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0 2024-09-17T23:31:39.319 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image cluster2 mirror image_10 2024-09-17T23:31:39.319 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:31:39.319 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:31:39.320 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_10 2024-09-17T23:31:39.320 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_10 2024-09-17T23:31:39.370 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 rm mirror/image_10 2024-09-17T23:31:39.485 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:31:39.481+0000 7ff501c2b640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55d142f43520 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-17T23:31:39.485 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:31:39.482+0000 7ff508c39640 -1 rbd::mirror::ImageReplayer: 0x55d14358a280 [3/2dc39d68-70b6-4566-beb0-8904894734d5] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-17T23:31:40.271 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-17T23:31:40.266+0000 7f8ef9635640 -1 reset not still connected to 0x55e56421cf70 2024-09-17T23:31:40.273 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-17T23:31:40.271+0000 7f8ef9635640 -1 reset not still connected to 0x55e56421d380 2024-09-17T23:31:40.273 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-17T23:31:40.271+0000 7f8ef9635640 -1 reset not still connected to 0x55e564ce7e10 2024-09-17T23:31:40.273 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-17T23:31:40.271+0000 7f8ef9635640 -1 reset not still connected to 0x55e56521fba0 2024-09-17T23:31:40.273 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-17T23:31:40.271+0000 7f8ef9635640 -1 reset not still connected to 0x55e56556d040 2024-09-17T23:31:40.273 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-17T23:31:40.271+0000 7f8ef9635640 -1 reset not still connected to 0x55e56644f6c0 2024-09-17T23:31:40.273 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-17T23:31:40.271+0000 7f8ef9635640 -1 reset not still connected to 0x55e56a8d6b60 2024-09-17T23:31:40.273 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-17T23:31:40.271+0000 7f8ef9635640 -1 reset not still connected to 0x55e56b1d6750 2024-09-17T23:31:40.274 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-17T23:31:40.271+0000 7f8ef9635640 -1 reset not still connected to 0x55e56b1d6a90 2024-09-17T23:31:40.274 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-17T23:31:40.271+0000 7f8ef9635640 -1 reset not still connected to 0x55e56b1d6b60 2024-09-17T23:31:40.274 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-17T23:31:40.271+0000 7f8ef9635640 -1 reset not still connected to 0x55e56b1d6d00 2024-09-17T23:31:40.274 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-17T23:31:40.271+0000 7f8ef9635640 -1 reset not still connected to 0x55e56b1d6dd0 2024-09-17T23:31:40.274 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-17T23:31:40.271+0000 7f8ef9635640 -1 reset not still connected to 0x55e56b1d7110 2024-09-17T23:31:40.274 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-17T23:31:40.271+0000 7f8ef9635640 -1 reset not still connected to 0x55e56b1d71e0 2024-09-17T23:31:40.274 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-17T23:31:40.271+0000 7f8ef9635640 -1 reset not still connected to 0x55e56b1d75f0 2024-09-17T23:31:41.434 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-17T23:31:41.437 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:31:41.438 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:31:41.438 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_11 2024-09-17T23:31:41.438 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_snapshot cluster2 mirror image_11 snap 2024-09-17T23:31:41.438 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:31:41.438 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:31:41.438 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_11 2024-09-17T23:31:41.438 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap=snap 2024-09-17T23:31:41.438 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap rm mirror/image_11@snap 2024-09-17T23:31:42.168 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing snap: 100% complete...done. 2024-09-17T23:31:42.205 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image_retry cluster2 mirror image_11 2024-09-17T23:31:42.205 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:31:42.205 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:31:42.205 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_11 2024-09-17T23:31:42.205 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-17T23:31:42.205 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0 2024-09-17T23:31:42.206 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image cluster2 mirror image_11 2024-09-17T23:31:42.206 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:31:42.206 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:31:42.206 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_11 2024-09-17T23:31:42.206 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_11 2024-09-17T23:31:42.254 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 rm mirror/image_11 2024-09-17T23:31:42.370 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:31:42.368+0000 7ff50242c640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55d142f436c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-17T23:31:42.370 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:31:42.368+0000 7ff508c39640 -1 rbd::mirror::ImageReplayer: 0x55d14358ac80 [3/bd10d2f7-c8b2-4a32-94d9-b2af3d68bf2d] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-17T23:31:44.441 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-17T23:31:44.445 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:31:44.445 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:31:44.445 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_12 2024-09-17T23:31:44.445 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_snapshot cluster2 mirror image_12 snap 2024-09-17T23:31:44.446 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:31:44.446 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:31:44.446 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_12 2024-09-17T23:31:44.446 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap=snap 2024-09-17T23:31:44.446 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap rm mirror/image_12@snap 2024-09-17T23:31:45.319 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing snap: 100% complete...done. 2024-09-17T23:31:45.346 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image_retry cluster2 mirror image_12 2024-09-17T23:31:45.346 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:31:45.347 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:31:45.347 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_12 2024-09-17T23:31:45.347 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-17T23:31:45.347 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0 2024-09-17T23:31:45.353 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image cluster2 mirror image_12 2024-09-17T23:31:45.354 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:31:45.354 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:31:45.354 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_12 2024-09-17T23:31:45.354 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_12 2024-09-17T23:31:45.401 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 rm mirror/image_12 2024-09-17T23:31:45.523 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:31:45.521+0000 7ff50242c640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55d142f43860 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-17T23:31:45.523 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:31:45.521+0000 7ff508438640 -1 rbd::mirror::ImageReplayer: 0x55d14358af00 [3/1e208d5a-c3ae-4b3c-bb32-593db1dbda33] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-17T23:31:47.310 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-17T23:31:47.314 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:31:47.314 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:31:47.314 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_13 2024-09-17T23:31:47.314 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_snapshot cluster2 mirror image_13 snap 2024-09-17T23:31:47.314 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:31:47.314 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:31:47.315 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_13 2024-09-17T23:31:47.315 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap=snap 2024-09-17T23:31:47.315 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap rm mirror/image_13@snap 2024-09-17T23:31:48.328 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing snap: 100% complete...done. 2024-09-17T23:31:48.351 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image_retry cluster2 mirror image_13 2024-09-17T23:31:48.351 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:31:48.351 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:31:48.351 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_13 2024-09-17T23:31:48.351 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-17T23:31:48.351 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0 2024-09-17T23:31:48.352 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image cluster2 mirror image_13 2024-09-17T23:31:48.352 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:31:48.352 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:31:48.352 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_13 2024-09-17T23:31:48.353 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_13 2024-09-17T23:31:48.410 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 rm mirror/image_13 2024-09-17T23:31:48.524 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:31:48.522+0000 7ff501c2b640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55d142f43a00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-17T23:31:48.524 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:31:48.522+0000 7ff508438640 -1 rbd::mirror::ImageReplayer: 0x55d14358b180 [3/9ebe7c6f-3aa1-48f9-82b3-c20461625f84] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-17T23:31:50.472 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-17T23:31:50.475 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:31:50.476 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:31:50.476 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_14 2024-09-17T23:31:50.476 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_snapshot cluster2 mirror image_14 snap 2024-09-17T23:31:50.476 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:31:50.476 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:31:50.476 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_14 2024-09-17T23:31:50.476 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap=snap 2024-09-17T23:31:50.476 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap rm mirror/image_14@snap 2024-09-17T23:31:51.349 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing snap: 100% complete...done. 2024-09-17T23:31:51.381 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image_retry cluster2 mirror image_14 2024-09-17T23:31:51.381 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:31:51.381 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:31:51.381 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_14 2024-09-17T23:31:51.381 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-17T23:31:51.381 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0 2024-09-17T23:31:51.382 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image cluster2 mirror image_14 2024-09-17T23:31:51.382 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:31:51.382 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:31:51.382 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_14 2024-09-17T23:31:51.382 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_14 2024-09-17T23:31:51.433 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 rm mirror/image_14 2024-09-17T23:31:51.547 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:31:51.545+0000 7ff501c2b640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55d142f43ba0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-17T23:31:51.547 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:31:51.545+0000 7ff508438640 -1 rbd::mirror::ImageReplayer: 0x55d14358a500 [3/115aba0c-a419-49de-b089-0afc43656b9c] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-17T23:31:53.297 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-17T23:31:53.301 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:31:53.301 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:31:53.301 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_15 2024-09-17T23:31:53.301 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_snapshot cluster2 mirror image_15 snap 2024-09-17T23:31:53.301 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:31:53.301 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:31:53.301 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_15 2024-09-17T23:31:53.301 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap=snap 2024-09-17T23:31:53.301 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap rm mirror/image_15@snap 2024-09-17T23:31:54.351 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing snap: 100% complete...done. 2024-09-17T23:31:54.384 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image_retry cluster2 mirror image_15 2024-09-17T23:31:54.385 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:31:54.385 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:31:54.385 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_15 2024-09-17T23:31:54.385 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-17T23:31:54.385 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0 2024-09-17T23:31:54.386 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image cluster2 mirror image_15 2024-09-17T23:31:54.386 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:31:54.386 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:31:54.386 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_15 2024-09-17T23:31:54.386 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_15 2024-09-17T23:31:54.435 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 rm mirror/image_15 2024-09-17T23:31:54.547 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:31:54.545+0000 7ff50242c640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55d142f43d40 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-17T23:31:54.547 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:31:54.545+0000 7ff508438640 -1 rbd::mirror::ImageReplayer: 0x55d14358b400 [3/c5adf338-7077-4024-ab02-6a785842239c] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-17T23:31:55.968 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-09-17T23:31:55.968 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-09-17T23:31:55.968 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2024-09-17T23:31:55.969 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi038.stderr:2024-09-17T23:31:55.967+0000 7f8008fe9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 57715) UID: 1000 2024-09-17T23:31:56.491 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-17T23:31:56.495 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:31:56.495 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:31:56.495 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_16 2024-09-17T23:31:56.495 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_snapshot cluster2 mirror image_16 snap 2024-09-17T23:31:56.495 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:31:56.495 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:31:56.495 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_16 2024-09-17T23:31:56.495 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap=snap 2024-09-17T23:31:56.495 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap rm mirror/image_16@snap 2024-09-17T23:31:57.182 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing snap: 100% complete...done. 2024-09-17T23:31:57.201 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image_retry cluster2 mirror image_16 2024-09-17T23:31:57.201 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:31:57.201 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:31:57.202 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_16 2024-09-17T23:31:57.202 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-17T23:31:57.202 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0 2024-09-17T23:31:57.202 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image cluster2 mirror image_16 2024-09-17T23:31:57.202 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:31:57.203 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:31:57.203 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_16 2024-09-17T23:31:57.203 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_16 2024-09-17T23:31:57.252 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 rm mirror/image_16 2024-09-17T23:31:57.362 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:31:57.360+0000 7ff50242c640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55d142d19520 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-17T23:31:57.362 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:31:57.360+0000 7ff508438640 -1 rbd::mirror::ImageReplayer: 0x55d14358b680 [3/b0d8f9e8-4e5b-45cd-b488-325c8908f543] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-17T23:31:59.316 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-17T23:31:59.319 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:31:59.320 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:31:59.320 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_17 2024-09-17T23:31:59.320 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_snapshot cluster2 mirror image_17 snap 2024-09-17T23:31:59.320 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:31:59.320 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:31:59.320 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_17 2024-09-17T23:31:59.320 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap=snap 2024-09-17T23:31:59.320 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap rm mirror/image_17@snap 2024-09-17T23:32:00.371 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing snap: 100% complete...done. 2024-09-17T23:32:00.397 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image_retry cluster2 mirror image_17 2024-09-17T23:32:00.397 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:32:00.397 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:32:00.397 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_17 2024-09-17T23:32:00.397 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-17T23:32:00.398 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0 2024-09-17T23:32:00.398 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image cluster2 mirror image_17 2024-09-17T23:32:00.398 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:32:00.399 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:32:00.399 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_17 2024-09-17T23:32:00.399 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_17 2024-09-17T23:32:00.452 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 rm mirror/image_17 2024-09-17T23:32:00.567 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:32:00.564+0000 7ff501c2b640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55d142d196c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-17T23:32:00.567 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:32:00.564+0000 7ff508438640 -1 rbd::mirror::ImageReplayer: 0x55d14358b900 [3/1c42b8ab-b2d2-4388-b7b5-b3e796c582f2] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-17T23:32:02.312 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-17T23:32:02.316 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:32:02.316 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:32:02.316 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_18 2024-09-17T23:32:02.316 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_snapshot cluster2 mirror image_18 snap 2024-09-17T23:32:02.316 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:32:02.316 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:32:02.316 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_18 2024-09-17T23:32:02.316 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap=snap 2024-09-17T23:32:02.316 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap rm mirror/image_18@snap 2024-09-17T23:32:03.405 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing snap: 100% complete...done. 2024-09-17T23:32:03.433 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image_retry cluster2 mirror image_18 2024-09-17T23:32:03.433 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:32:03.433 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:32:03.433 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_18 2024-09-17T23:32:03.434 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-17T23:32:03.434 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0 2024-09-17T23:32:03.434 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image cluster2 mirror image_18 2024-09-17T23:32:03.434 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:32:03.435 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:32:03.435 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_18 2024-09-17T23:32:03.435 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_18 2024-09-17T23:32:03.485 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 rm mirror/image_18 2024-09-17T23:32:03.600 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:32:03.598+0000 7ff501c2b640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55d142f32b60 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-17T23:32:03.601 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:32:03.598+0000 7ff508438640 -1 rbd::mirror::ImageReplayer: 0x55d142730000 [3/82669132-31fe-4405-bf71-3a31515c863e] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-17T23:32:03.969 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-09-17T23:32:03.969 INFO:teuthology.orchestra.run:waiting for 600 2024-09-17T23:32:03.970 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-09-17T23:32:03.970 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-09-17T23:32:03.970 DEBUG:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-09-17T23:32:03.974 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-09-17T23:32:03.974 INFO:tasks.rbd_mirror_thrash:waiting for 106 secs before thrashing 2024-09-17T23:32:05.547 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-17T23:32:05.550 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:32:05.550 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:32:05.551 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_19 2024-09-17T23:32:05.551 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_snapshot cluster2 mirror image_19 snap 2024-09-17T23:32:05.551 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:32:05.551 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:32:05.551 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_19 2024-09-17T23:32:05.551 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap=snap 2024-09-17T23:32:05.551 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap rm mirror/image_19@snap 2024-09-17T23:32:06.414 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing snap: 100% complete...done. 2024-09-17T23:32:06.451 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image_retry cluster2 mirror image_19 2024-09-17T23:32:06.451 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:32:06.451 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:32:06.451 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_19 2024-09-17T23:32:06.451 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-17T23:32:06.451 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0 2024-09-17T23:32:06.452 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image cluster2 mirror image_19 2024-09-17T23:32:06.452 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:32:06.452 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:32:06.452 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_19 2024-09-17T23:32:06.452 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_19 2024-09-17T23:32:06.501 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 rm mirror/image_19 2024-09-17T23:32:06.617 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:32:06.615+0000 7ff50242c640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55d142f4eea0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-17T23:32:06.617 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:32:06.615+0000 7ff508c39640 -1 rbd::mirror::ImageReplayer: 0x55d142730280 [3/263f2d9e-807d-4be1-b51d-f2bd7fb7a5d6] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-17T23:32:08.313 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-17T23:32:08.316 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:32:08.316 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:32:08.316 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_20 2024-09-17T23:32:08.316 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_snapshot cluster2 mirror image_20 snap 2024-09-17T23:32:08.316 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:32:08.316 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:32:08.316 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_20 2024-09-17T23:32:08.316 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap=snap 2024-09-17T23:32:08.317 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap rm mirror/image_20@snap 2024-09-17T23:32:09.188 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing snap: 100% complete...done. 2024-09-17T23:32:09.204 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image_retry cluster2 mirror image_20 2024-09-17T23:32:09.204 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:32:09.204 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:32:09.204 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_20 2024-09-17T23:32:09.204 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-17T23:32:09.205 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0 2024-09-17T23:32:09.205 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image cluster2 mirror image_20 2024-09-17T23:32:09.205 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:32:09.205 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:32:09.206 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_20 2024-09-17T23:32:09.206 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_20 2024-09-17T23:32:09.454 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 rm mirror/image_20 2024-09-17T23:32:09.567 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:32:09.565+0000 7ff501c2b640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55d142d18b60 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-17T23:32:09.568 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:32:09.565+0000 7ff508438640 -1 rbd::mirror::ImageReplayer: 0x55d142730500 [3/04cb45cc-a535-47bd-b9a9-46bf5ca4fe67] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-17T23:32:11.334 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-17T23:32:11.338 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:32:11.338 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:32:11.338 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_21 2024-09-17T23:32:11.338 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_snapshot cluster2 mirror image_21 snap 2024-09-17T23:32:11.339 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:32:11.339 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:32:11.339 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_21 2024-09-17T23:32:11.339 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap=snap 2024-09-17T23:32:11.339 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap rm mirror/image_21@snap 2024-09-17T23:32:12.177 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing snap: 100% complete...done. 2024-09-17T23:32:12.211 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image_retry cluster2 mirror image_21 2024-09-17T23:32:12.211 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:32:12.211 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:32:12.211 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_21 2024-09-17T23:32:12.211 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-17T23:32:12.211 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0 2024-09-17T23:32:12.213 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image cluster2 mirror image_21 2024-09-17T23:32:12.213 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:32:12.213 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:32:12.213 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_21 2024-09-17T23:32:12.213 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_21 2024-09-17T23:32:12.265 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 rm mirror/image_21 2024-09-17T23:32:12.345 INFO:tasks.ceph.cluster2.osd.2.smithi038.stderr:2024-09-17T23:32:12.342+0000 7f1c4b04a640 -1 reset not still connected to 0x561cedd37ba0 2024-09-17T23:32:12.345 INFO:tasks.ceph.cluster2.osd.2.smithi038.stderr:2024-09-17T23:32:12.342+0000 7f1c4b04a640 -1 reset not still connected to 0x561cedd37c70 2024-09-17T23:32:12.345 INFO:tasks.ceph.cluster2.osd.2.smithi038.stderr:2024-09-17T23:32:12.342+0000 7f1c4b04a640 -1 reset not still connected to 0x561cedd37d40 2024-09-17T23:32:12.345 INFO:tasks.ceph.cluster2.osd.2.smithi038.stderr:2024-09-17T23:32:12.342+0000 7f1c4b04a640 -1 reset not still connected to 0x561cedd37e10 2024-09-17T23:32:12.389 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:32:12.387+0000 7ff50242c640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55d142f50340 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-17T23:32:12.389 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:32:12.387+0000 7ff508c39640 -1 rbd::mirror::ImageReplayer: 0x55d14358bb80 [3/7f670507-4ecc-426a-a941-5287c614d52d] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-17T23:32:14.345 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-17T23:32:14.348 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:32:14.349 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:32:14.349 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_22 2024-09-17T23:32:14.349 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_snapshot cluster2 mirror image_22 snap 2024-09-17T23:32:14.349 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:32:14.349 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:32:14.349 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_22 2024-09-17T23:32:14.349 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap=snap 2024-09-17T23:32:14.349 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap rm mirror/image_22@snap 2024-09-17T23:32:15.231 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing snap: 100% complete...done. 2024-09-17T23:32:15.253 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image_retry cluster2 mirror image_22 2024-09-17T23:32:15.253 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:32:15.253 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:32:15.253 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_22 2024-09-17T23:32:15.253 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-17T23:32:15.253 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0 2024-09-17T23:32:15.254 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image cluster2 mirror image_22 2024-09-17T23:32:15.254 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:32:15.254 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:32:15.254 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_22 2024-09-17T23:32:15.254 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_22 2024-09-17T23:32:15.304 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 rm mirror/image_22 2024-09-17T23:32:15.627 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:32:15.624+0000 7ff50242c640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55d142d18d00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-17T23:32:15.627 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:32:15.624+0000 7ff508438640 -1 rbd::mirror::ImageReplayer: 0x55d142731180 [3/1068bb88-a6ef-49c0-a048-fafa200170dc] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-17T23:32:17.319 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-17T23:32:17.322 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:32:17.322 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:32:17.322 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_23 2024-09-17T23:32:17.322 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_snapshot cluster2 mirror image_23 snap 2024-09-17T23:32:17.322 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:32:17.323 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:32:17.323 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_23 2024-09-17T23:32:17.323 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap=snap 2024-09-17T23:32:17.323 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap rm mirror/image_23@snap 2024-09-17T23:32:18.233 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing snap: 100% complete...done. 2024-09-17T23:32:18.266 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image_retry cluster2 mirror image_23 2024-09-17T23:32:18.266 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:32:18.266 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:32:18.266 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_23 2024-09-17T23:32:18.267 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-17T23:32:18.267 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0 2024-09-17T23:32:18.268 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image cluster2 mirror image_23 2024-09-17T23:32:18.268 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:32:18.268 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:32:18.268 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_23 2024-09-17T23:32:18.268 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_23 2024-09-17T23:32:18.317 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 rm mirror/image_23 2024-09-17T23:32:18.438 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:32:18.434+0000 7ff501c2b640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55d142f4fba0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-17T23:32:18.438 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:32:18.434+0000 7ff508c39640 -1 rbd::mirror::ImageReplayer: 0x55d142730780 [3/7b178615-290c-4f56-9926-be968e7bf83a] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-17T23:32:20.389 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-17T23:32:20.393 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:32:20.393 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:32:20.393 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_24 2024-09-17T23:32:20.393 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_snapshot cluster2 mirror image_24 snap 2024-09-17T23:32:20.393 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:32:20.393 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:32:20.393 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_24 2024-09-17T23:32:20.393 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap=snap 2024-09-17T23:32:20.393 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap rm mirror/image_24@snap 2024-09-17T23:32:21.265 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing snap: 100% complete...done. 2024-09-17T23:32:21.291 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image_retry cluster2 mirror image_24 2024-09-17T23:32:21.291 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:32:21.291 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:32:21.291 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_24 2024-09-17T23:32:21.291 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-17T23:32:21.291 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0 2024-09-17T23:32:21.293 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image cluster2 mirror image_24 2024-09-17T23:32:21.293 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:32:21.293 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:32:21.293 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_24 2024-09-17T23:32:21.293 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_24 2024-09-17T23:32:21.344 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 rm mirror/image_24 2024-09-17T23:32:21.457 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:32:21.455+0000 7ff501c2b640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55d142d18ea0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-17T23:32:21.458 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:32:21.455+0000 7ff508c39640 -1 rbd::mirror::ImageReplayer: 0x55d142731400 [3/de674b7e-029c-4272-9b26-d163572b5aea] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-17T23:32:23.312 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-17T23:32:23.316 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:32:23.316 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:32:23.316 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_25 2024-09-17T23:32:23.316 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_snapshot cluster2 mirror image_25 snap 2024-09-17T23:32:23.316 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:32:23.316 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:32:23.316 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_25 2024-09-17T23:32:23.316 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap=snap 2024-09-17T23:32:23.316 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap rm mirror/image_25@snap 2024-09-17T23:32:24.262 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing snap: 100% complete...done. 2024-09-17T23:32:24.286 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image_retry cluster2 mirror image_25 2024-09-17T23:32:24.286 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:32:24.287 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:32:24.287 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_25 2024-09-17T23:32:24.287 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-17T23:32:24.287 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0 2024-09-17T23:32:24.287 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image cluster2 mirror image_25 2024-09-17T23:32:24.288 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:32:24.288 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:32:24.288 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_25 2024-09-17T23:32:24.288 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_25 2024-09-17T23:32:24.347 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 rm mirror/image_25 2024-09-17T23:32:24.467 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:32:24.465+0000 7ff501c2b640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55d142d19040 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-17T23:32:24.467 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:32:24.465+0000 7ff508438640 -1 rbd::mirror::ImageReplayer: 0x55d142731680 [3/d504d61f-d7b1-44e8-a7fe-4d46bee040b2] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-17T23:32:26.403 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-17T23:32:26.406 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:32:26.406 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:32:26.406 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_26 2024-09-17T23:32:26.407 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_snapshot cluster2 mirror image_26 snap 2024-09-17T23:32:26.407 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:32:26.407 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:32:26.407 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_26 2024-09-17T23:32:26.407 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap=snap 2024-09-17T23:32:26.407 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap rm mirror/image_26@snap 2024-09-17T23:32:27.175 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing snap: 100% complete...done. 2024-09-17T23:32:27.209 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image_retry cluster2 mirror image_26 2024-09-17T23:32:27.209 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:32:27.209 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:32:27.209 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_26 2024-09-17T23:32:27.209 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-17T23:32:27.209 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0 2024-09-17T23:32:27.210 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image cluster2 mirror image_26 2024-09-17T23:32:27.210 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:32:27.210 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:32:27.210 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_26 2024-09-17T23:32:27.210 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_26 2024-09-17T23:32:27.258 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 rm mirror/image_26 2024-09-17T23:32:27.371 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:32:27.369+0000 7ff501c2b640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55d142d13380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-17T23:32:27.372 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:32:27.369+0000 7ff508438640 -1 rbd::mirror::ImageReplayer: 0x55d1438b2000 [3/abfb6ed3-8b33-429f-bf1d-a7fe92644887] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-17T23:32:29.428 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-17T23:32:29.431 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:32:29.432 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:32:29.432 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_27 2024-09-17T23:32:29.432 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_snapshot cluster2 mirror image_27 snap 2024-09-17T23:32:29.432 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:32:29.432 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:32:29.432 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_27 2024-09-17T23:32:29.432 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap=snap 2024-09-17T23:32:29.432 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap rm mirror/image_27@snap 2024-09-17T23:32:30.305 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing snap: 100% complete...done. 2024-09-17T23:32:30.327 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image_retry cluster2 mirror image_27 2024-09-17T23:32:30.327 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:32:30.327 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:32:30.327 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_27 2024-09-17T23:32:30.327 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-17T23:32:30.327 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0 2024-09-17T23:32:30.328 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image cluster2 mirror image_27 2024-09-17T23:32:30.328 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:32:30.328 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:32:30.328 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_27 2024-09-17T23:32:30.328 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_27 2024-09-17T23:32:30.379 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 rm mirror/image_27 2024-09-17T23:32:30.498 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:32:30.496+0000 7ff501c2b640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55d142f429c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-17T23:32:30.498 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:32:30.496+0000 7ff508c39640 -1 rbd::mirror::ImageReplayer: 0x55d142731900 [3/3340f84d-5843-4769-b7b7-cd6b8eee8192] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-17T23:32:32.322 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-17T23:32:32.326 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:32:32.326 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:32:32.326 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_28 2024-09-17T23:32:32.326 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_snapshot cluster2 mirror image_28 snap 2024-09-17T23:32:32.326 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:32:32.327 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:32:32.327 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_28 2024-09-17T23:32:32.327 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap=snap 2024-09-17T23:32:32.327 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap rm mirror/image_28@snap 2024-09-17T23:32:33.316 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing snap: 100% complete...done. 2024-09-17T23:32:33.339 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image_retry cluster2 mirror image_28 2024-09-17T23:32:33.339 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:32:33.339 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:32:33.339 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_28 2024-09-17T23:32:33.339 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-17T23:32:33.339 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0 2024-09-17T23:32:33.340 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image cluster2 mirror image_28 2024-09-17T23:32:33.340 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:32:33.340 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:32:33.340 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_28 2024-09-17T23:32:33.340 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_28 2024-09-17T23:32:33.398 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 rm mirror/image_28 2024-09-17T23:32:33.512 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:32:33.510+0000 7ff50242c640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55d142d13520 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-17T23:32:33.512 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:32:33.510+0000 7ff508438640 -1 rbd::mirror::ImageReplayer: 0x55d1438b2280 [3/aa5b5df7-2c98-4b35-a04c-18d3b14fb4a4] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-17T23:32:35.462 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-17T23:32:35.465 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:32:35.465 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:32:35.466 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_29 2024-09-17T23:32:35.466 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_snapshot cluster2 mirror image_29 snap 2024-09-17T23:32:35.466 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:32:35.466 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:32:35.466 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_29 2024-09-17T23:32:35.466 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap=snap 2024-09-17T23:32:35.466 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap rm mirror/image_29@snap 2024-09-17T23:32:36.334 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing snap: 100% complete...done. 2024-09-17T23:32:36.363 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image_retry cluster2 mirror image_29 2024-09-17T23:32:36.363 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:32:36.363 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:32:36.363 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_29 2024-09-17T23:32:36.363 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-17T23:32:36.363 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0 2024-09-17T23:32:36.364 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image cluster2 mirror image_29 2024-09-17T23:32:36.364 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:32:36.364 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:32:36.364 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_29 2024-09-17T23:32:36.364 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_29 2024-09-17T23:32:36.418 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 rm mirror/image_29 2024-09-17T23:32:36.540 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:32:36.538+0000 7ff50242c640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55d142d136c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-17T23:32:36.540 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:32:36.538+0000 7ff508438640 -1 rbd::mirror::ImageReplayer: 0x55d142731b80 [3/5ce3fe34-b38d-4dad-9e4f-b5cdc9405a96] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-17T23:32:38.316 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-17T23:32:38.320 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:32:38.320 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:32:38.320 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_30 2024-09-17T23:32:38.321 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_snapshot cluster2 mirror image_30 snap 2024-09-17T23:32:38.321 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:32:38.321 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:32:38.321 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_30 2024-09-17T23:32:38.321 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap=snap 2024-09-17T23:32:38.321 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap rm mirror/image_30@snap 2024-09-17T23:32:39.184 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing snap: 100% complete...done. 2024-09-17T23:32:39.216 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image_retry cluster2 mirror image_30 2024-09-17T23:32:39.216 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:32:39.216 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:32:39.216 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_30 2024-09-17T23:32:39.216 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-17T23:32:39.216 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0 2024-09-17T23:32:39.217 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image cluster2 mirror image_30 2024-09-17T23:32:39.217 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:32:39.217 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:32:39.217 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_30 2024-09-17T23:32:39.217 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_30 2024-09-17T23:32:39.267 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 rm mirror/image_30 2024-09-17T23:32:39.391 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:32:39.389+0000 7ff50242c640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55d142f42b60 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-17T23:32:39.391 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:32:39.389+0000 7ff508c39640 -1 rbd::mirror::ImageReplayer: 0x55d1438b2500 [3/c1039ee2-f5c2-417c-8ea6-8094c3f49b33] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-17T23:32:41.327 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-17T23:32:41.331 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:32:41.331 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:32:41.331 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_31 2024-09-17T23:32:41.331 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_snapshot cluster2 mirror image_31 snap 2024-09-17T23:32:41.331 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:32:41.331 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:32:41.331 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_31 2024-09-17T23:32:41.331 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap=snap 2024-09-17T23:32:41.331 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap rm mirror/image_31@snap 2024-09-17T23:32:42.195 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing snap: 100% complete...done. 2024-09-17T23:32:42.217 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image_retry cluster2 mirror image_31 2024-09-17T23:32:42.217 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:32:42.217 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:32:42.218 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_31 2024-09-17T23:32:42.218 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-17T23:32:42.218 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0 2024-09-17T23:32:42.218 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image cluster2 mirror image_31 2024-09-17T23:32:42.219 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:32:42.219 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:32:42.219 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_31 2024-09-17T23:32:42.219 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_31 2024-09-17T23:32:42.267 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 rm mirror/image_31 2024-09-17T23:32:42.380 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:32:42.378+0000 7ff50242c640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55d142d13860 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-17T23:32:42.380 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:32:42.378+0000 7ff508438640 -1 rbd::mirror::ImageReplayer: 0x55d1438b2c80 [3/c8ba8555-2fc3-405e-b6a5-7ff5d6a4334c] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-17T23:32:44.347 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-17T23:32:44.352 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:32:44.353 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:32:44.353 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_32 2024-09-17T23:32:44.353 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_snapshot cluster2 mirror image_32 snap 2024-09-17T23:32:44.353 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:32:44.353 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:32:44.353 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_32 2024-09-17T23:32:44.353 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap=snap 2024-09-17T23:32:44.353 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap rm mirror/image_32@snap 2024-09-17T23:32:45.213 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing snap: 100% complete...done. 2024-09-17T23:32:45.249 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image_retry cluster2 mirror image_32 2024-09-17T23:32:45.249 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:32:45.249 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:32:45.249 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_32 2024-09-17T23:32:45.249 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-17T23:32:45.249 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0 2024-09-17T23:32:45.250 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image cluster2 mirror image_32 2024-09-17T23:32:45.250 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:32:45.250 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:32:45.250 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_32 2024-09-17T23:32:45.250 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_32 2024-09-17T23:32:45.300 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 rm mirror/image_32 2024-09-17T23:32:45.432 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:32:45.430+0000 7ff50242c640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55d142d13a00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-17T23:32:45.433 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:32:45.430+0000 7ff508438640 -1 rbd::mirror::ImageReplayer: 0x55d1438b2f00 [3/c47737c4-4126-4b7d-8eff-21c600ab88e3] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-17T23:32:47.322 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-17T23:32:47.326 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:32:47.326 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:32:47.326 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_33 2024-09-17T23:32:47.326 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_snapshot cluster2 mirror image_33 snap 2024-09-17T23:32:47.326 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:32:47.326 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:32:47.326 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_33 2024-09-17T23:32:47.326 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap=snap 2024-09-17T23:32:47.326 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap rm mirror/image_33@snap 2024-09-17T23:32:48.222 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing snap: 100% complete...done. 2024-09-17T23:32:48.253 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image_retry cluster2 mirror image_33 2024-09-17T23:32:48.253 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:32:48.254 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:32:48.254 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_33 2024-09-17T23:32:48.254 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-17T23:32:48.254 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0 2024-09-17T23:32:48.255 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image cluster2 mirror image_33 2024-09-17T23:32:48.255 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:32:48.255 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:32:48.256 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_33 2024-09-17T23:32:48.256 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_33 2024-09-17T23:32:48.307 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 rm mirror/image_33 2024-09-17T23:32:48.427 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:32:48.424+0000 7ff50242c640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55d142f44820 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-17T23:32:48.427 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:32:48.424+0000 7ff508c39640 -1 rbd::mirror::ImageReplayer: 0x55d1438b3180 [3/d3b01291-4a3c-45c8-9b7e-71a70aa71cd3] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-17T23:32:50.372 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-17T23:32:50.375 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:32:50.375 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:32:50.376 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_34 2024-09-17T23:32:50.376 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_snapshot cluster2 mirror image_34 snap 2024-09-17T23:32:50.376 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:32:50.376 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:32:50.376 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_34 2024-09-17T23:32:50.376 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap=snap 2024-09-17T23:32:50.376 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap rm mirror/image_34@snap 2024-09-17T23:32:51.238 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing snap: 100% complete...done. 2024-09-17T23:32:51.271 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image_retry cluster2 mirror image_34 2024-09-17T23:32:51.271 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:32:51.271 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:32:51.271 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_34 2024-09-17T23:32:51.271 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-17T23:32:51.271 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0 2024-09-17T23:32:51.272 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image cluster2 mirror image_34 2024-09-17T23:32:51.272 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:32:51.272 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:32:51.272 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_34 2024-09-17T23:32:51.272 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_34 2024-09-17T23:32:51.324 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 rm mirror/image_34 2024-09-17T23:32:51.442 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:32:51.440+0000 7ff50242c640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55d142d13ba0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-17T23:32:51.442 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:32:51.440+0000 7ff508438640 -1 rbd::mirror::ImageReplayer: 0x55d1438b3400 [3/78b7bbf1-8bd9-4fb6-b1bb-58a0b0412b92] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-17T23:32:52.863 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.62482499722281) > weight (0.25) 2024-09-17T23:32:52.863 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.3953986029337707) > weight (0.25) 2024-09-17T23:32:52.863 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.31395494975441196) > weight (0.25) 2024-09-17T23:32:52.863 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.47328908470894016) > weight (0.25) 2024-09-17T23:32:52.863 INFO:tasks.rbd_mirror_thrash:waiting for 60 secs before thrashing 2024-09-17T23:32:53.315 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-17T23:32:53.319 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:32:53.319 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:32:53.320 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_35 2024-09-17T23:32:53.320 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_snapshot cluster2 mirror image_35 snap 2024-09-17T23:32:53.320 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:32:53.320 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:32:53.320 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_35 2024-09-17T23:32:53.320 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap=snap 2024-09-17T23:32:53.320 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap rm mirror/image_35@snap 2024-09-17T23:32:53.411 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-17T23:32:53.408+0000 7f8ef9635640 -1 reset not still connected to 0x55e56b1d7110 2024-09-17T23:32:53.411 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-17T23:32:53.408+0000 7f8ef9635640 -1 reset not still connected to 0x55e56b1d75f0 2024-09-17T23:32:54.243 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing snap: 100% complete...done. 2024-09-17T23:32:54.270 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image_retry cluster2 mirror image_35 2024-09-17T23:32:54.270 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:32:54.270 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:32:54.270 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_35 2024-09-17T23:32:54.270 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-17T23:32:54.270 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0 2024-09-17T23:32:54.272 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image cluster2 mirror image_35 2024-09-17T23:32:54.272 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:32:54.272 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:32:54.272 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_35 2024-09-17T23:32:54.272 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_35 2024-09-17T23:32:54.327 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 rm mirror/image_35 2024-09-17T23:32:54.444 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:32:54.442+0000 7ff501c2b640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55d142f47a00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-17T23:32:54.444 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:32:54.442+0000 7ff508438640 -1 rbd::mirror::ImageReplayer: 0x55d1438b2780 [3/b8f54ad1-6d1a-42f7-821f-2ff1a07ca075] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-17T23:32:56.378 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-17T23:32:56.381 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:32:56.382 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:32:56.382 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_36 2024-09-17T23:32:56.382 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_snapshot cluster2 mirror image_36 snap 2024-09-17T23:32:56.382 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:32:56.382 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:32:56.382 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_36 2024-09-17T23:32:56.382 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap=snap 2024-09-17T23:32:56.382 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap rm mirror/image_36@snap 2024-09-17T23:32:57.204 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing snap: 100% complete...done. 2024-09-17T23:32:57.224 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image_retry cluster2 mirror image_36 2024-09-17T23:32:57.224 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:32:57.224 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:32:57.225 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_36 2024-09-17T23:32:57.225 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-17T23:32:57.225 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0 2024-09-17T23:32:57.225 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image cluster2 mirror image_36 2024-09-17T23:32:57.225 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:32:57.226 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:32:57.226 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_36 2024-09-17T23:32:57.226 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_36 2024-09-17T23:32:57.273 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 rm mirror/image_36 2024-09-17T23:32:57.388 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:32:57.386+0000 7ff501c2b640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55d142f329c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-17T23:32:57.388 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:32:57.386+0000 7ff508c39640 -1 rbd::mirror::ImageReplayer: 0x55d1438b3680 [3/2e12bc1d-2eb3-49b1-b5a5-e5c5013f32d1] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-17T23:32:59.404 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-17T23:32:59.408 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:32:59.408 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:32:59.408 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_37 2024-09-17T23:32:59.408 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_snapshot cluster2 mirror image_37 snap 2024-09-17T23:32:59.409 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:32:59.409 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:32:59.409 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_37 2024-09-17T23:32:59.409 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap=snap 2024-09-17T23:32:59.409 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap rm mirror/image_37@snap 2024-09-17T23:33:00.277 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing snap: 100% complete...done. 2024-09-17T23:33:00.300 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image_retry cluster2 mirror image_37 2024-09-17T23:33:00.300 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:33:00.300 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:00.301 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_37 2024-09-17T23:33:00.301 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-17T23:33:00.301 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0 2024-09-17T23:33:00.301 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image cluster2 mirror image_37 2024-09-17T23:33:00.301 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:33:00.301 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:00.302 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_37 2024-09-17T23:33:00.302 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_37 2024-09-17T23:33:00.355 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 rm mirror/image_37 2024-09-17T23:33:00.471 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:33:00.468+0000 7ff50242c640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55d142f3e820 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-17T23:33:00.471 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:33:00.468+0000 7ff508c39640 -1 rbd::mirror::ImageReplayer: 0x55d1438b3900 [3/ccc8bbca-2b19-4cff-8cb4-7770b06359dc] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-17T23:33:02.326 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-17T23:33:02.329 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:33:02.329 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:33:02.329 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_38 2024-09-17T23:33:02.330 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_snapshot cluster2 mirror image_38 snap 2024-09-17T23:33:02.330 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:33:02.330 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:02.330 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_38 2024-09-17T23:33:02.330 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap=snap 2024-09-17T23:33:02.330 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap rm mirror/image_38@snap 2024-09-17T23:33:03.308 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing snap: 100% complete...done. 2024-09-17T23:33:03.330 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image_retry cluster2 mirror image_38 2024-09-17T23:33:03.330 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:33:03.331 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:03.331 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_38 2024-09-17T23:33:03.331 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-17T23:33:03.331 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0 2024-09-17T23:33:03.331 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image cluster2 mirror image_38 2024-09-17T23:33:03.332 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:33:03.332 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:03.332 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_38 2024-09-17T23:33:03.332 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_38 2024-09-17T23:33:03.584 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 rm mirror/image_38 2024-09-17T23:33:03.707 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:33:03.705+0000 7ff50242c640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55d142f4f1e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-17T23:33:03.708 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:33:03.705+0000 7ff508c39640 -1 rbd::mirror::ImageReplayer: 0x55d1438b2a00 [3/37d0dace-6324-49a7-9dcc-0827dcbac606] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-17T23:33:05.446 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-17T23:33:05.449 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:33:05.449 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:33:05.449 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_39 2024-09-17T23:33:05.450 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_snapshot cluster2 mirror image_39 snap 2024-09-17T23:33:05.450 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:33:05.450 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:05.450 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_39 2024-09-17T23:33:05.450 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap=snap 2024-09-17T23:33:05.450 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap rm mirror/image_39@snap 2024-09-17T23:33:06.315 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing snap: 100% complete...done. 2024-09-17T23:33:06.333 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image_retry cluster2 mirror image_39 2024-09-17T23:33:06.333 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:33:06.333 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:06.333 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_39 2024-09-17T23:33:06.333 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-17T23:33:06.333 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0 2024-09-17T23:33:06.334 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image cluster2 mirror image_39 2024-09-17T23:33:06.334 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:33:06.334 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:06.334 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_39 2024-09-17T23:33:06.334 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_39 2024-09-17T23:33:06.383 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 rm mirror/image_39 2024-09-17T23:33:06.501 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:33:06.499+0000 7ff501c2b640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55d142d13d40 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-17T23:33:06.502 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:33:06.499+0000 7ff508c39640 -1 rbd::mirror::ImageReplayer: 0x55d143242000 [3/a0492a7c-0945-424f-9742-2a83199e6bda] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-17T23:33:08.321 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-17T23:33:08.325 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:33:08.325 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:33:08.325 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_40 2024-09-17T23:33:08.325 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_snapshot cluster2 mirror image_40 snap 2024-09-17T23:33:08.325 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:33:08.325 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:08.325 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_40 2024-09-17T23:33:08.325 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap=snap 2024-09-17T23:33:08.326 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap rm mirror/image_40@snap 2024-09-17T23:33:09.204 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing snap: 100% complete...done. 2024-09-17T23:33:09.231 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image_retry cluster2 mirror image_40 2024-09-17T23:33:09.231 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:33:09.231 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:09.231 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_40 2024-09-17T23:33:09.231 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-17T23:33:09.231 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0 2024-09-17T23:33:09.232 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image cluster2 mirror image_40 2024-09-17T23:33:09.232 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:33:09.232 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:09.233 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_40 2024-09-17T23:33:09.233 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_40 2024-09-17T23:33:09.284 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 rm mirror/image_40 2024-09-17T23:33:09.399 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:33:09.395+0000 7ff50242c640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55d142f501a0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-17T23:33:09.399 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:33:09.395+0000 7ff508c39640 -1 rbd::mirror::ImageReplayer: 0x55d1438b3b80 [3/2fcc3df0-03d9-483f-9f1f-787ec2efd74d] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-17T23:33:11.351 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-17T23:33:11.355 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:33:11.355 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:33:11.355 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_41 2024-09-17T23:33:11.355 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_snapshot cluster2 mirror image_41 snap 2024-09-17T23:33:11.355 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:33:11.355 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:11.355 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_41 2024-09-17T23:33:11.355 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap=snap 2024-09-17T23:33:11.355 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap rm mirror/image_41@snap 2024-09-17T23:33:12.196 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing snap: 100% complete...done. 2024-09-17T23:33:12.225 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image_retry cluster2 mirror image_41 2024-09-17T23:33:12.225 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:33:12.225 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:12.225 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_41 2024-09-17T23:33:12.225 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-17T23:33:12.225 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0 2024-09-17T23:33:12.226 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image cluster2 mirror image_41 2024-09-17T23:33:12.226 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:33:12.226 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:12.226 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_41 2024-09-17T23:33:12.226 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_41 2024-09-17T23:33:12.277 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 rm mirror/image_41 2024-09-17T23:33:12.392 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:33:12.389+0000 7ff50242c640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55d142d18000 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-17T23:33:12.392 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:33:12.389+0000 7ff508438640 -1 rbd::mirror::ImageReplayer: 0x55d143242f00 [3/7a67405f-a922-4db6-a34d-aef043e8bddf] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-17T23:33:14.362 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-17T23:33:14.366 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:33:14.366 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:33:14.366 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_42 2024-09-17T23:33:14.366 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_snapshot cluster2 mirror image_42 snap 2024-09-17T23:33:14.366 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:33:14.366 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:14.366 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_42 2024-09-17T23:33:14.366 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap=snap 2024-09-17T23:33:14.367 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap rm mirror/image_42@snap 2024-09-17T23:33:15.247 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing snap: 100% complete...done. 2024-09-17T23:33:15.275 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image_retry cluster2 mirror image_42 2024-09-17T23:33:15.275 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:33:15.275 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:15.275 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_42 2024-09-17T23:33:15.275 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-17T23:33:15.275 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0 2024-09-17T23:33:15.276 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image cluster2 mirror image_42 2024-09-17T23:33:15.277 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:33:15.277 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:15.277 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_42 2024-09-17T23:33:15.277 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_42 2024-09-17T23:33:15.328 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 rm mirror/image_42 2024-09-17T23:33:15.442 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:33:15.440+0000 7ff501c2b640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55d142d181a0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-17T23:33:15.442 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:33:15.440+0000 7ff508c39640 -1 rbd::mirror::ImageReplayer: 0x55d143242500 [3/515d2514-6060-40dc-8860-1c0b3ca01bb9] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-17T23:33:17.325 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-17T23:33:17.329 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:33:17.329 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:33:17.329 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_43 2024-09-17T23:33:17.329 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_snapshot cluster2 mirror image_43 snap 2024-09-17T23:33:17.329 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:33:17.329 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:17.329 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_43 2024-09-17T23:33:17.329 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap=snap 2024-09-17T23:33:17.330 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap rm mirror/image_43@snap 2024-09-17T23:33:18.271 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing snap: 100% complete...done. 2024-09-17T23:33:18.295 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image_retry cluster2 mirror image_43 2024-09-17T23:33:18.295 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:33:18.295 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:18.295 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_43 2024-09-17T23:33:18.295 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-17T23:33:18.296 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0 2024-09-17T23:33:18.297 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image cluster2 mirror image_43 2024-09-17T23:33:18.297 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:33:18.297 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:18.297 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_43 2024-09-17T23:33:18.297 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_43 2024-09-17T23:33:18.550 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 rm mirror/image_43 2024-09-17T23:33:18.673 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:33:18.671+0000 7ff501c2b640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55d142d1e340 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-17T23:33:18.673 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:33:18.671+0000 7ff508c39640 -1 rbd::mirror::ImageReplayer: 0x55d143242780 [3/65d764ca-56da-405c-b68b-d13fb878db55] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-17T23:33:20.415 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-17T23:33:20.418 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:33:20.418 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:33:20.418 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_44 2024-09-17T23:33:20.418 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_snapshot cluster2 mirror image_44 snap 2024-09-17T23:33:20.418 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:33:20.418 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:20.419 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_44 2024-09-17T23:33:20.419 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap=snap 2024-09-17T23:33:20.419 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap rm mirror/image_44@snap 2024-09-17T23:33:21.296 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing snap: 100% complete...done. 2024-09-17T23:33:21.324 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image_retry cluster2 mirror image_44 2024-09-17T23:33:21.324 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:33:21.324 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:21.324 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_44 2024-09-17T23:33:21.324 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-17T23:33:21.324 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0 2024-09-17T23:33:21.325 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image cluster2 mirror image_44 2024-09-17T23:33:21.325 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:33:21.325 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:21.325 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_44 2024-09-17T23:33:21.326 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_44 2024-09-17T23:33:21.377 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 rm mirror/image_44 2024-09-17T23:33:21.491 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:33:21.489+0000 7ff50242c640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55d142d1e4e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-17T23:33:21.492 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:33:21.489+0000 7ff508438640 -1 rbd::mirror::ImageReplayer: 0x55d143242a00 [3/1c859b2d-e379-40d7-bc46-427fb4a14245] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-17T23:33:23.321 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-17T23:33:23.325 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:33:23.325 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:33:23.325 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_45 2024-09-17T23:33:23.325 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_snapshot cluster2 mirror image_45 snap 2024-09-17T23:33:23.325 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:33:23.325 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:23.325 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_45 2024-09-17T23:33:23.326 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap=snap 2024-09-17T23:33:23.326 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap rm mirror/image_45@snap 2024-09-17T23:33:24.293 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing snap: 100% complete...done. 2024-09-17T23:33:24.323 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image_retry cluster2 mirror image_45 2024-09-17T23:33:24.324 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:33:24.324 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:24.324 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_45 2024-09-17T23:33:24.324 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-17T23:33:24.324 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0 2024-09-17T23:33:24.325 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image cluster2 mirror image_45 2024-09-17T23:33:24.325 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:33:24.325 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:24.325 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_45 2024-09-17T23:33:24.325 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_45 2024-09-17T23:33:24.378 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 rm mirror/image_45 2024-09-17T23:33:24.502 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:33:24.500+0000 7ff50242c640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55d142d19380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-17T23:33:24.503 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:33:24.500+0000 7ff508c39640 -1 rbd::mirror::ImageReplayer: 0x55d143242c80 [3/a44d8381-af32-4227-90fa-0531c242a7f7] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-17T23:33:26.442 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-17T23:33:26.446 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:33:26.446 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:33:26.446 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_46 2024-09-17T23:33:26.446 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_snapshot cluster2 mirror image_46 snap 2024-09-17T23:33:26.446 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:33:26.446 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:26.446 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_46 2024-09-17T23:33:26.447 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap=snap 2024-09-17T23:33:26.447 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap rm mirror/image_46@snap 2024-09-17T23:33:27.209 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing snap: 100% complete...done. 2024-09-17T23:33:27.227 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image_retry cluster2 mirror image_46 2024-09-17T23:33:27.227 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:33:27.227 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:27.227 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_46 2024-09-17T23:33:27.227 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-17T23:33:27.227 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0 2024-09-17T23:33:27.228 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image cluster2 mirror image_46 2024-09-17T23:33:27.229 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:33:27.229 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:27.229 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_46 2024-09-17T23:33:27.229 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_46 2024-09-17T23:33:27.278 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 rm mirror/image_46 2024-09-17T23:33:27.401 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:33:27.399+0000 7ff50242c640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55d142d18680 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-17T23:33:27.401 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:33:27.399+0000 7ff508c39640 -1 rbd::mirror::ImageReplayer: 0x55d1427a4000 [3/bb8f076b-e4d3-47ce-9e64-72dd50f71318] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-17T23:33:29.472 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-17T23:33:29.476 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:33:29.476 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:33:29.476 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_47 2024-09-17T23:33:29.476 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_snapshot cluster2 mirror image_47 snap 2024-09-17T23:33:29.476 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:33:29.476 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:29.476 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_47 2024-09-17T23:33:29.476 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap=snap 2024-09-17T23:33:29.476 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap rm mirror/image_47@snap 2024-09-17T23:33:30.342 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing snap: 100% complete...done. 2024-09-17T23:33:30.374 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image_retry cluster2 mirror image_47 2024-09-17T23:33:30.375 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:33:30.375 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:30.375 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_47 2024-09-17T23:33:30.375 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-17T23:33:30.375 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0 2024-09-17T23:33:30.376 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image cluster2 mirror image_47 2024-09-17T23:33:30.376 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:33:30.376 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:30.376 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_47 2024-09-17T23:33:30.376 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_47 2024-09-17T23:33:30.426 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 rm mirror/image_47 2024-09-17T23:33:30.549 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:33:30.547+0000 7ff501c2b640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55d142d07860 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-17T23:33:30.549 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:33:30.547+0000 7ff508438640 -1 rbd::mirror::ImageReplayer: 0x55d14318a780 [3/04e3e2c8-dba3-4aac-a2db-a1e1fd0c90d3] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-17T23:33:32.326 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-17T23:33:32.330 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:33:32.330 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:33:32.330 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_48 2024-09-17T23:33:32.330 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_snapshot cluster2 mirror image_48 snap 2024-09-17T23:33:32.330 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:33:32.330 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:32.330 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_48 2024-09-17T23:33:32.330 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap=snap 2024-09-17T23:33:32.330 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap rm mirror/image_48@snap 2024-09-17T23:33:33.360 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing snap: 100% complete...done. 2024-09-17T23:33:33.390 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image_retry cluster2 mirror image_48 2024-09-17T23:33:33.390 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:33:33.390 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:33.390 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_48 2024-09-17T23:33:33.390 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-17T23:33:33.390 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0 2024-09-17T23:33:33.391 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image cluster2 mirror image_48 2024-09-17T23:33:33.391 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:33:33.392 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:33.392 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_48 2024-09-17T23:33:33.392 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_48 2024-09-17T23:33:33.444 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 rm mirror/image_48 2024-09-17T23:33:33.561 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:33:33.559+0000 7ff501c2b640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55d142d189c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-17T23:33:33.561 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:33:33.559+0000 7ff508c39640 -1 rbd::mirror::ImageReplayer: 0x55d143243180 [3/38a2f93b-1926-44fa-95d5-32067464b867] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-17T23:33:35.500 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-17T23:33:35.504 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:33:35.504 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:33:35.504 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_49 2024-09-17T23:33:35.504 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_snapshot cluster2 mirror image_49 snap 2024-09-17T23:33:35.504 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:33:35.505 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:35.505 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_49 2024-09-17T23:33:35.505 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap=snap 2024-09-17T23:33:35.505 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap rm mirror/image_49@snap 2024-09-17T23:33:36.384 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing snap: 100% complete...done. 2024-09-17T23:33:36.409 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image_retry cluster2 mirror image_49 2024-09-17T23:33:36.409 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:33:36.410 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:36.410 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_49 2024-09-17T23:33:36.410 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-17T23:33:36.410 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0 2024-09-17T23:33:36.410 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image cluster2 mirror image_49 2024-09-17T23:33:36.410 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:33:36.410 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:36.411 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_49 2024-09-17T23:33:36.411 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_49 2024-09-17T23:33:36.459 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 rm mirror/image_49 2024-09-17T23:33:36.581 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:33:36.579+0000 7ff50242c640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55d142d18820 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-17T23:33:36.581 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:33:36.579+0000 7ff508438640 -1 rbd::mirror::ImageReplayer: 0x55d1438cc000 [3/61b5b6dd-6f04-4689-abba-edcd8570ecec] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-17T23:33:38.327 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-17T23:33:38.331 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:33:38.331 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:33:38.331 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_50 2024-09-17T23:33:38.331 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_snapshot cluster2 mirror image_50 snap 2024-09-17T23:33:38.331 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:33:38.331 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:38.332 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_50 2024-09-17T23:33:38.332 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local snap=snap 2024-09-17T23:33:38.332 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap rm mirror/image_50@snap 2024-09-17T23:33:39.207 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing snap: 100% complete...done. 2024-09-17T23:33:39.232 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image_retry cluster2 mirror image_50 2024-09-17T23:33:39.232 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:33:39.232 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:39.232 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_50 2024-09-17T23:33:39.232 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-17T23:33:39.232 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0 2024-09-17T23:33:39.233 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image cluster2 mirror image_50 2024-09-17T23:33:39.233 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:33:39.234 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:39.234 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_50 2024-09-17T23:33:39.234 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_50 2024-09-17T23:33:39.285 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 rm mirror/image_50 2024-09-17T23:33:39.402 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:33:39.400+0000 7ff50242c640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55d142d19d40 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-17T23:33:39.402 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:33:39.400+0000 7ff508438640 -1 rbd::mirror::ImageReplayer: 0x55d1438cc280 [3/865bd16a-a0e6-4a3a-bb53-8569e6a216ea] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-17T23:33:41.344 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-17T23:33:41.348 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:33:41.348 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ testlog 'TEST: image deletions should propagate' 2024-09-17T23:33:41.348 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ tee -a /tmp/tmp.s5MEFDgsej/rbd-mirror.test.log 2024-09-17T23:33:41.348 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ date '+%F %T' 2024-09-17T23:33:41.349 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ echo 2024-09-17 23:33:41 TEST: image deletions should propagate 2024-09-17T23:33:41.349 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:2024-09-17 23:33:41 TEST: image deletions should propagate 2024-09-17T23:33:41.349 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_pool_images cluster1 mirror 0 2024-09-17T23:33:41.349 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:33:41.349 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:41.349 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_count=0 2024-09-17T23:33:41.349 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:33:41.349 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local count 2024-09-17T23:33:41.350 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local last_count=0 2024-09-17T23:33:41.350 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ true 2024-09-17T23:33:41.350 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ seq 1 40 2024-09-17T23:33:41.351 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in `seq 1 40` 2024-09-17T23:33:41.351 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 1 -ne 1 2024-09-17T23:33:41.351 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2024-09-17T23:33:41.351 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ grep 'images: ' 2024-09-17T23:33:41.351 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cut '-d ' -f 2 2024-09-17T23:33:41.386 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: failed to query services: (13) Permission denied 2024-09-17T23:33:41.390 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ count=0 2024-09-17T23:33:41.390 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 0 = 0 2024-09-17T23:33:41.390 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:33:41.390 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_pool_healthy cluster1 mirror 0 2024-09-17T23:33:41.390 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:33:41.390 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:41.390 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:33:41.390 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state 2024-09-17T23:33:41.391 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ seq 1 40 2024-09-17T23:33:41.391 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in `seq 1 40` 2024-09-17T23:33:41.392 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 1 -ne 1 2024-09-17T23:33:41.392 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2024-09-17T23:33:41.392 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ grep 'image health:' 2024-09-17T23:33:41.392 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cut '-d ' -f 3 2024-09-17T23:33:41.425 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: failed to query services: (13) Permission denied 2024-09-17T23:33:41.428 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ state=OK 2024-09-17T23:33:41.429 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test OK = ERROR 2024-09-17T23:33:41.429 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test OK = OK 2024-09-17T23:33:41.429 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:33:41.429 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ seq 1 50 2024-09-17T23:33:41.430 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:33:41.430 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_1 2024-09-17T23:33:41.430 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_present cluster1 mirror image_1 deleted 2024-09-17T23:33:41.431 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:33:41.431 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:41.431 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_1 2024-09-17T23:33:41.431 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=deleted 2024-09-17T23:33:41.431 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:33:41.431 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:33:41.431 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:33:41.431 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_1 2024-09-17T23:33:41.431 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:33:41.431 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:33:41.431 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_1 2024-09-17T23:33:41.431 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-09-17T23:33:41.431 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:33:41.463 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_1: (2) No such file or directory 2024-09-17T23:33:41.466 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image_id= 2024-09-17T23:33:41.466 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-17T23:33:41.466 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.1 2024-09-17T23:33:41.568 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_present cluster1 mirror image_1 deleted '' 2024-09-17T23:33:41.568 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:33:41.568 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:41.568 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_1 2024-09-17T23:33:41.568 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=deleted 2024-09-17T23:33:41.568 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:33:41.568 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=deleted 2024-09-17T23:33:41.569 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_image_id 2024-09-17T23:33:41.569 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_1 2024-09-17T23:33:41.569 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:33:41.569 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:33:41.569 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_1 2024-09-17T23:33:41.569 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-09-17T23:33:41.569 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:33:41.601 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_1: (2) No such file or directory 2024-09-17T23:33:41.605 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_image_id= 2024-09-17T23:33:41.605 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:33:41.606 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test deleted = deleted 2024-09-17T23:33:41.606 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:33:41.606 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:33:41.606 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_2 2024-09-17T23:33:41.606 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_present cluster1 mirror image_2 deleted 2024-09-17T23:33:41.606 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:33:41.606 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:41.606 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_2 2024-09-17T23:33:41.606 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=deleted 2024-09-17T23:33:41.606 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:33:41.606 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:33:41.606 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:33:41.606 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_2 2024-09-17T23:33:41.606 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:33:41.607 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:33:41.607 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_2 2024-09-17T23:33:41.607 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-09-17T23:33:41.607 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:33:41.638 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_2: (2) No such file or directory 2024-09-17T23:33:41.641 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image_id= 2024-09-17T23:33:41.641 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-17T23:33:41.641 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.1 2024-09-17T23:33:41.743 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_present cluster1 mirror image_2 deleted '' 2024-09-17T23:33:41.743 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:33:41.744 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:41.744 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_2 2024-09-17T23:33:41.744 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=deleted 2024-09-17T23:33:41.744 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:33:41.744 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=deleted 2024-09-17T23:33:41.744 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_image_id 2024-09-17T23:33:41.744 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_2 2024-09-17T23:33:41.744 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:33:41.744 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:33:41.744 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_2 2024-09-17T23:33:41.744 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-09-17T23:33:41.744 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:33:41.777 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_2: (2) No such file or directory 2024-09-17T23:33:41.781 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_image_id= 2024-09-17T23:33:41.781 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:33:41.781 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test deleted = deleted 2024-09-17T23:33:41.782 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:33:41.782 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:33:41.782 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_3 2024-09-17T23:33:41.782 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_present cluster1 mirror image_3 deleted 2024-09-17T23:33:41.782 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:33:41.782 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:41.782 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_3 2024-09-17T23:33:41.782 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=deleted 2024-09-17T23:33:41.782 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:33:41.782 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:33:41.782 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:33:41.782 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_3 2024-09-17T23:33:41.782 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:33:41.782 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:33:41.782 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_3 2024-09-17T23:33:41.783 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-09-17T23:33:41.783 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:33:41.816 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_3: (2) No such file or directory 2024-09-17T23:33:41.819 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image_id= 2024-09-17T23:33:41.819 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-17T23:33:41.819 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.1 2024-09-17T23:33:41.921 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_present cluster1 mirror image_3 deleted '' 2024-09-17T23:33:41.921 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:33:41.921 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:41.921 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_3 2024-09-17T23:33:41.921 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=deleted 2024-09-17T23:33:41.921 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:33:41.921 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=deleted 2024-09-17T23:33:41.921 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_image_id 2024-09-17T23:33:41.921 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_3 2024-09-17T23:33:41.921 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:33:41.922 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:33:41.922 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_3 2024-09-17T23:33:41.922 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-09-17T23:33:41.922 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:33:41.955 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_3: (2) No such file or directory 2024-09-17T23:33:41.959 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_image_id= 2024-09-17T23:33:41.959 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:33:41.959 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test deleted = deleted 2024-09-17T23:33:41.959 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:33:41.959 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:33:41.959 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_4 2024-09-17T23:33:41.959 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_present cluster1 mirror image_4 deleted 2024-09-17T23:33:41.959 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:33:41.959 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:41.959 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_4 2024-09-17T23:33:41.959 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=deleted 2024-09-17T23:33:41.959 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:33:41.960 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:33:41.960 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:33:41.960 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_4 2024-09-17T23:33:41.960 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:33:41.960 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:33:41.960 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_4 2024-09-17T23:33:41.960 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-09-17T23:33:41.960 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:33:41.992 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_4: (2) No such file or directory 2024-09-17T23:33:41.995 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image_id= 2024-09-17T23:33:41.995 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-17T23:33:41.996 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.1 2024-09-17T23:33:42.097 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_present cluster1 mirror image_4 deleted '' 2024-09-17T23:33:42.097 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:33:42.098 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:42.098 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_4 2024-09-17T23:33:42.098 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=deleted 2024-09-17T23:33:42.098 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:33:42.098 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=deleted 2024-09-17T23:33:42.098 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_image_id 2024-09-17T23:33:42.098 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_4 2024-09-17T23:33:42.098 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:33:42.098 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:33:42.098 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_4 2024-09-17T23:33:42.098 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-09-17T23:33:42.098 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:33:42.131 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_4: (2) No such file or directory 2024-09-17T23:33:42.135 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_image_id= 2024-09-17T23:33:42.135 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:33:42.136 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test deleted = deleted 2024-09-17T23:33:42.136 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:33:42.136 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:33:42.136 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_5 2024-09-17T23:33:42.136 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_present cluster1 mirror image_5 deleted 2024-09-17T23:33:42.136 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:33:42.136 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:42.136 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_5 2024-09-17T23:33:42.136 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=deleted 2024-09-17T23:33:42.136 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:33:42.136 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:33:42.136 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:33:42.137 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_5 2024-09-17T23:33:42.137 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:33:42.137 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:33:42.137 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_5 2024-09-17T23:33:42.137 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-09-17T23:33:42.137 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:33:42.168 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_5: (2) No such file or directory 2024-09-17T23:33:42.172 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image_id= 2024-09-17T23:33:42.172 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-17T23:33:42.172 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.1 2024-09-17T23:33:42.273 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_present cluster1 mirror image_5 deleted '' 2024-09-17T23:33:42.274 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:33:42.274 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:42.274 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_5 2024-09-17T23:33:42.274 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=deleted 2024-09-17T23:33:42.274 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:33:42.274 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=deleted 2024-09-17T23:33:42.274 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_image_id 2024-09-17T23:33:42.274 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_5 2024-09-17T23:33:42.274 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:33:42.274 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:33:42.274 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_5 2024-09-17T23:33:42.274 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-09-17T23:33:42.275 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:33:42.308 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_5: (2) No such file or directory 2024-09-17T23:33:42.312 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_image_id= 2024-09-17T23:33:42.312 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:33:42.312 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test deleted = deleted 2024-09-17T23:33:42.312 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:33:42.312 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:33:42.312 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_6 2024-09-17T23:33:42.312 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_present cluster1 mirror image_6 deleted 2024-09-17T23:33:42.312 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:33:42.312 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:42.313 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_6 2024-09-17T23:33:42.313 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=deleted 2024-09-17T23:33:42.313 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:33:42.313 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:33:42.313 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:33:42.313 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_6 2024-09-17T23:33:42.313 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:33:42.313 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:33:42.313 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_6 2024-09-17T23:33:42.313 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-09-17T23:33:42.313 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:33:42.345 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_6: (2) No such file or directory 2024-09-17T23:33:42.348 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image_id= 2024-09-17T23:33:42.349 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-17T23:33:42.349 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.1 2024-09-17T23:33:42.450 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_present cluster1 mirror image_6 deleted '' 2024-09-17T23:33:42.450 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:33:42.450 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:42.451 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_6 2024-09-17T23:33:42.451 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=deleted 2024-09-17T23:33:42.451 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:33:42.451 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=deleted 2024-09-17T23:33:42.451 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_image_id 2024-09-17T23:33:42.451 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_6 2024-09-17T23:33:42.451 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:33:42.451 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:33:42.451 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_6 2024-09-17T23:33:42.451 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-09-17T23:33:42.451 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:33:42.485 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_6: (2) No such file or directory 2024-09-17T23:33:42.490 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_image_id= 2024-09-17T23:33:42.490 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:33:42.490 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test deleted = deleted 2024-09-17T23:33:42.490 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:33:42.490 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:33:42.490 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_7 2024-09-17T23:33:42.490 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_present cluster1 mirror image_7 deleted 2024-09-17T23:33:42.490 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:33:42.490 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:42.490 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_7 2024-09-17T23:33:42.490 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=deleted 2024-09-17T23:33:42.491 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:33:42.491 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:33:42.491 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:33:42.491 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_7 2024-09-17T23:33:42.491 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:33:42.491 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:33:42.491 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_7 2024-09-17T23:33:42.491 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-09-17T23:33:42.491 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:33:42.525 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_7: (2) No such file or directory 2024-09-17T23:33:42.528 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image_id= 2024-09-17T23:33:42.528 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-17T23:33:42.528 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.1 2024-09-17T23:33:42.630 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_present cluster1 mirror image_7 deleted '' 2024-09-17T23:33:42.630 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:33:42.630 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:42.630 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_7 2024-09-17T23:33:42.630 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=deleted 2024-09-17T23:33:42.631 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:33:42.631 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=deleted 2024-09-17T23:33:42.631 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_image_id 2024-09-17T23:33:42.631 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_7 2024-09-17T23:33:42.631 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:33:42.631 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:33:42.631 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_7 2024-09-17T23:33:42.631 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-09-17T23:33:42.631 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:33:42.665 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_7: (2) No such file or directory 2024-09-17T23:33:42.668 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_image_id= 2024-09-17T23:33:42.669 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:33:42.669 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test deleted = deleted 2024-09-17T23:33:42.669 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:33:42.669 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:33:42.669 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_8 2024-09-17T23:33:42.669 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_present cluster1 mirror image_8 deleted 2024-09-17T23:33:42.669 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:33:42.669 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:42.669 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_8 2024-09-17T23:33:42.669 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=deleted 2024-09-17T23:33:42.669 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:33:42.669 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:33:42.669 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:33:42.669 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_8 2024-09-17T23:33:42.670 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:33:42.670 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:33:42.670 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_8 2024-09-17T23:33:42.670 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-09-17T23:33:42.670 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:33:42.904 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_8: (2) No such file or directory 2024-09-17T23:33:42.908 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image_id= 2024-09-17T23:33:42.908 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-17T23:33:42.908 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.1 2024-09-17T23:33:43.010 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_present cluster1 mirror image_8 deleted '' 2024-09-17T23:33:43.010 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:33:43.010 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:43.010 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_8 2024-09-17T23:33:43.010 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=deleted 2024-09-17T23:33:43.010 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:33:43.010 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=deleted 2024-09-17T23:33:43.010 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_image_id 2024-09-17T23:33:43.010 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_8 2024-09-17T23:33:43.010 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:33:43.011 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:33:43.011 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_8 2024-09-17T23:33:43.011 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-09-17T23:33:43.011 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:33:43.044 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_8: (2) No such file or directory 2024-09-17T23:33:43.048 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_image_id= 2024-09-17T23:33:43.048 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:33:43.049 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test deleted = deleted 2024-09-17T23:33:43.049 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:33:43.049 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:33:43.049 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_9 2024-09-17T23:33:43.049 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_present cluster1 mirror image_9 deleted 2024-09-17T23:33:43.049 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:33:43.049 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:43.049 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_9 2024-09-17T23:33:43.049 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=deleted 2024-09-17T23:33:43.049 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:33:43.049 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:33:43.049 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:33:43.049 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_9 2024-09-17T23:33:43.049 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:33:43.050 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:33:43.050 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_9 2024-09-17T23:33:43.050 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-09-17T23:33:43.050 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:33:43.082 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_9: (2) No such file or directory 2024-09-17T23:33:43.085 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image_id= 2024-09-17T23:33:43.085 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-17T23:33:43.086 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.1 2024-09-17T23:33:43.187 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_present cluster1 mirror image_9 deleted '' 2024-09-17T23:33:43.188 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:33:43.188 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:43.188 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_9 2024-09-17T23:33:43.188 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=deleted 2024-09-17T23:33:43.188 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:33:43.188 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=deleted 2024-09-17T23:33:43.188 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_image_id 2024-09-17T23:33:43.188 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_9 2024-09-17T23:33:43.188 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:33:43.188 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:33:43.188 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_9 2024-09-17T23:33:43.189 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-09-17T23:33:43.189 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:33:43.221 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_9: (2) No such file or directory 2024-09-17T23:33:43.225 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_image_id= 2024-09-17T23:33:43.225 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:33:43.225 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test deleted = deleted 2024-09-17T23:33:43.225 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:33:43.225 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:33:43.226 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_10 2024-09-17T23:33:43.226 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_present cluster1 mirror image_10 deleted 2024-09-17T23:33:43.226 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:33:43.226 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:43.226 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_10 2024-09-17T23:33:43.226 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=deleted 2024-09-17T23:33:43.226 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:33:43.226 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:33:43.226 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:33:43.226 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_10 2024-09-17T23:33:43.226 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:33:43.226 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:33:43.226 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_10 2024-09-17T23:33:43.227 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-09-17T23:33:43.227 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:33:43.460 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_10: (2) No such file or directory 2024-09-17T23:33:43.463 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image_id= 2024-09-17T23:33:43.463 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-17T23:33:43.463 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.1 2024-09-17T23:33:43.565 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_present cluster1 mirror image_10 deleted '' 2024-09-17T23:33:43.565 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:33:43.565 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:43.565 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_10 2024-09-17T23:33:43.565 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=deleted 2024-09-17T23:33:43.565 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:33:43.565 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=deleted 2024-09-17T23:33:43.565 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_image_id 2024-09-17T23:33:43.565 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_10 2024-09-17T23:33:43.566 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:33:43.566 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:33:43.566 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_10 2024-09-17T23:33:43.566 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-09-17T23:33:43.566 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:33:43.600 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_10: (2) No such file or directory 2024-09-17T23:33:43.604 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_image_id= 2024-09-17T23:33:43.604 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:33:43.604 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test deleted = deleted 2024-09-17T23:33:43.604 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:33:43.604 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:33:43.604 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_11 2024-09-17T23:33:43.604 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_present cluster1 mirror image_11 deleted 2024-09-17T23:33:43.604 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:33:43.604 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:43.605 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_11 2024-09-17T23:33:43.605 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=deleted 2024-09-17T23:33:43.605 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:33:43.605 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:33:43.605 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:33:43.605 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_11 2024-09-17T23:33:43.605 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:33:43.605 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:33:43.605 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_11 2024-09-17T23:33:43.605 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_11 2024-09-17T23:33:43.605 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:33:43.638 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_11: (2) No such file or directory 2024-09-17T23:33:43.641 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image_id= 2024-09-17T23:33:43.641 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-17T23:33:43.642 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.1 2024-09-17T23:33:43.743 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_present cluster1 mirror image_11 deleted '' 2024-09-17T23:33:43.743 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:33:43.744 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:43.744 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_11 2024-09-17T23:33:43.744 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=deleted 2024-09-17T23:33:43.744 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:33:43.744 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=deleted 2024-09-17T23:33:43.744 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_image_id 2024-09-17T23:33:43.744 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_11 2024-09-17T23:33:43.744 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:33:43.744 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:33:43.744 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_11 2024-09-17T23:33:43.744 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_11 2024-09-17T23:33:43.744 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:33:43.777 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_11: (2) No such file or directory 2024-09-17T23:33:43.782 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_image_id= 2024-09-17T23:33:43.782 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:33:43.782 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test deleted = deleted 2024-09-17T23:33:43.782 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:33:43.782 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:33:43.782 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_12 2024-09-17T23:33:43.782 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_present cluster1 mirror image_12 deleted 2024-09-17T23:33:43.782 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:33:43.783 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:43.783 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_12 2024-09-17T23:33:43.783 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=deleted 2024-09-17T23:33:43.783 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:33:43.783 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:33:43.783 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:33:43.784 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_12 2024-09-17T23:33:43.784 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:33:43.784 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:33:43.784 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_12 2024-09-17T23:33:43.784 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_12 2024-09-17T23:33:43.784 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:33:43.815 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_12: (2) No such file or directory 2024-09-17T23:33:43.818 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image_id= 2024-09-17T23:33:43.818 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-17T23:33:43.818 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.1 2024-09-17T23:33:43.920 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_present cluster1 mirror image_12 deleted '' 2024-09-17T23:33:43.920 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:33:43.920 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:43.920 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_12 2024-09-17T23:33:43.920 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=deleted 2024-09-17T23:33:43.920 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:33:43.921 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=deleted 2024-09-17T23:33:43.921 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_image_id 2024-09-17T23:33:43.921 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_12 2024-09-17T23:33:43.921 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:33:43.921 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:33:43.921 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_12 2024-09-17T23:33:43.921 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_12 2024-09-17T23:33:43.921 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:33:43.955 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_12: (2) No such file or directory 2024-09-17T23:33:43.959 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_image_id= 2024-09-17T23:33:43.960 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:33:43.960 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test deleted = deleted 2024-09-17T23:33:43.960 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:33:43.960 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:33:43.960 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_13 2024-09-17T23:33:43.960 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_present cluster1 mirror image_13 deleted 2024-09-17T23:33:43.960 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:33:43.960 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:43.960 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_13 2024-09-17T23:33:43.960 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=deleted 2024-09-17T23:33:43.960 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:33:43.960 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:33:43.961 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:33:43.961 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_13 2024-09-17T23:33:43.961 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:33:43.961 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:33:43.961 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_13 2024-09-17T23:33:43.961 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_13 2024-09-17T23:33:43.961 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:33:43.992 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_13: (2) No such file or directory 2024-09-17T23:33:43.995 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image_id= 2024-09-17T23:33:43.995 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-17T23:33:43.995 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.1 2024-09-17T23:33:44.097 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_present cluster1 mirror image_13 deleted '' 2024-09-17T23:33:44.097 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:33:44.098 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:44.098 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_13 2024-09-17T23:33:44.098 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=deleted 2024-09-17T23:33:44.098 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:33:44.098 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=deleted 2024-09-17T23:33:44.098 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_image_id 2024-09-17T23:33:44.098 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_13 2024-09-17T23:33:44.098 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:33:44.098 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:33:44.098 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_13 2024-09-17T23:33:44.099 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_13 2024-09-17T23:33:44.099 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:33:44.131 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_13: (2) No such file or directory 2024-09-17T23:33:44.135 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_image_id= 2024-09-17T23:33:44.135 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:33:44.135 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test deleted = deleted 2024-09-17T23:33:44.135 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:33:44.135 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:33:44.135 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_14 2024-09-17T23:33:44.135 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_present cluster1 mirror image_14 deleted 2024-09-17T23:33:44.135 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:33:44.136 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:44.136 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_14 2024-09-17T23:33:44.136 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=deleted 2024-09-17T23:33:44.136 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:33:44.136 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:33:44.136 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:33:44.136 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_14 2024-09-17T23:33:44.136 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:33:44.136 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:33:44.136 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_14 2024-09-17T23:33:44.136 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_14 2024-09-17T23:33:44.136 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:33:44.168 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_14: (2) No such file or directory 2024-09-17T23:33:44.171 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image_id= 2024-09-17T23:33:44.171 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-17T23:33:44.171 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.1 2024-09-17T23:33:44.273 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_present cluster1 mirror image_14 deleted '' 2024-09-17T23:33:44.273 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:33:44.273 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:44.273 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_14 2024-09-17T23:33:44.273 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=deleted 2024-09-17T23:33:44.273 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:33:44.273 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=deleted 2024-09-17T23:33:44.274 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_image_id 2024-09-17T23:33:44.274 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_14 2024-09-17T23:33:44.274 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:33:44.274 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:33:44.274 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_14 2024-09-17T23:33:44.274 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_14 2024-09-17T23:33:44.274 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:33:44.307 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_14: (2) No such file or directory 2024-09-17T23:33:44.311 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_image_id= 2024-09-17T23:33:44.311 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:33:44.311 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test deleted = deleted 2024-09-17T23:33:44.311 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:33:44.311 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:33:44.311 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_15 2024-09-17T23:33:44.311 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_present cluster1 mirror image_15 deleted 2024-09-17T23:33:44.311 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:33:44.312 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:44.312 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_15 2024-09-17T23:33:44.312 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=deleted 2024-09-17T23:33:44.312 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:33:44.312 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:33:44.312 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:33:44.312 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_15 2024-09-17T23:33:44.312 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:33:44.312 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:33:44.312 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_15 2024-09-17T23:33:44.312 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_15 2024-09-17T23:33:44.312 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:33:44.344 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_15: (2) No such file or directory 2024-09-17T23:33:44.347 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image_id= 2024-09-17T23:33:44.347 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-17T23:33:44.347 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.1 2024-09-17T23:33:44.449 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_present cluster1 mirror image_15 deleted '' 2024-09-17T23:33:44.449 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:33:44.449 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:44.449 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_15 2024-09-17T23:33:44.449 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=deleted 2024-09-17T23:33:44.449 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:33:44.449 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=deleted 2024-09-17T23:33:44.450 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_image_id 2024-09-17T23:33:44.450 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_15 2024-09-17T23:33:44.450 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:33:44.450 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:33:44.450 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_15 2024-09-17T23:33:44.450 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_15 2024-09-17T23:33:44.450 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:33:44.484 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_15: (2) No such file or directory 2024-09-17T23:33:44.489 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_image_id= 2024-09-17T23:33:44.489 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:33:44.489 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test deleted = deleted 2024-09-17T23:33:44.489 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:33:44.489 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:33:44.489 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_16 2024-09-17T23:33:44.489 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_present cluster1 mirror image_16 deleted 2024-09-17T23:33:44.489 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:33:44.489 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:44.489 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_16 2024-09-17T23:33:44.490 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=deleted 2024-09-17T23:33:44.490 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:33:44.490 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:33:44.490 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:33:44.490 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_16 2024-09-17T23:33:44.490 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:33:44.490 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:33:44.490 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_16 2024-09-17T23:33:44.490 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_16 2024-09-17T23:33:44.490 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:33:44.522 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_16: (2) No such file or directory 2024-09-17T23:33:44.525 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image_id= 2024-09-17T23:33:44.525 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-17T23:33:44.525 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.1 2024-09-17T23:33:44.627 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_present cluster1 mirror image_16 deleted '' 2024-09-17T23:33:44.627 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:33:44.627 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:44.627 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_16 2024-09-17T23:33:44.627 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=deleted 2024-09-17T23:33:44.627 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:33:44.627 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=deleted 2024-09-17T23:33:44.627 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_image_id 2024-09-17T23:33:44.627 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_16 2024-09-17T23:33:44.627 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:33:44.628 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:33:44.628 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_16 2024-09-17T23:33:44.628 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_16 2024-09-17T23:33:44.628 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:33:44.660 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_16: (2) No such file or directory 2024-09-17T23:33:44.664 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_image_id= 2024-09-17T23:33:44.664 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:33:44.664 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test deleted = deleted 2024-09-17T23:33:44.664 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:33:44.664 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:33:44.664 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_17 2024-09-17T23:33:44.664 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_present cluster1 mirror image_17 deleted 2024-09-17T23:33:44.664 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:33:44.665 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:44.665 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_17 2024-09-17T23:33:44.665 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=deleted 2024-09-17T23:33:44.665 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:33:44.665 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:33:44.665 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:33:44.665 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_17 2024-09-17T23:33:44.665 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:33:44.665 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:33:44.665 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_17 2024-09-17T23:33:44.665 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_17 2024-09-17T23:33:44.665 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:33:44.697 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_17: (2) No such file or directory 2024-09-17T23:33:44.700 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image_id= 2024-09-17T23:33:44.700 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-17T23:33:44.700 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.1 2024-09-17T23:33:44.802 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_present cluster1 mirror image_17 deleted '' 2024-09-17T23:33:44.802 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:33:44.802 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:44.802 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_17 2024-09-17T23:33:44.802 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=deleted 2024-09-17T23:33:44.802 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:33:44.802 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=deleted 2024-09-17T23:33:44.802 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_image_id 2024-09-17T23:33:44.802 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_17 2024-09-17T23:33:44.802 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:33:44.802 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:33:44.803 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_17 2024-09-17T23:33:44.803 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_17 2024-09-17T23:33:44.803 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:33:44.835 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_17: (2) No such file or directory 2024-09-17T23:33:44.839 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_image_id= 2024-09-17T23:33:44.839 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:33:44.839 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test deleted = deleted 2024-09-17T23:33:44.839 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:33:44.839 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:33:44.839 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_18 2024-09-17T23:33:44.839 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_present cluster1 mirror image_18 deleted 2024-09-17T23:33:44.839 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:33:44.840 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:44.840 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_18 2024-09-17T23:33:44.840 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=deleted 2024-09-17T23:33:44.840 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:33:44.840 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:33:44.840 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:33:44.840 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_18 2024-09-17T23:33:44.840 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:33:44.840 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:33:44.840 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_18 2024-09-17T23:33:44.840 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_18 2024-09-17T23:33:44.840 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:33:44.874 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_18: (2) No such file or directory 2024-09-17T23:33:44.877 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image_id= 2024-09-17T23:33:44.877 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-17T23:33:44.877 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.1 2024-09-17T23:33:44.979 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_present cluster1 mirror image_18 deleted '' 2024-09-17T23:33:44.979 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:33:44.979 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:44.979 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_18 2024-09-17T23:33:44.979 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=deleted 2024-09-17T23:33:44.979 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:33:44.979 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=deleted 2024-09-17T23:33:44.979 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_image_id 2024-09-17T23:33:44.979 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_18 2024-09-17T23:33:44.980 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:33:44.980 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:33:44.980 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_18 2024-09-17T23:33:44.980 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_18 2024-09-17T23:33:44.980 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:33:45.012 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_18: (2) No such file or directory 2024-09-17T23:33:45.017 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_image_id= 2024-09-17T23:33:45.017 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:33:45.017 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test deleted = deleted 2024-09-17T23:33:45.017 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:33:45.017 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:33:45.017 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_19 2024-09-17T23:33:45.017 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_present cluster1 mirror image_19 deleted 2024-09-17T23:33:45.017 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:33:45.017 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:45.017 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_19 2024-09-17T23:33:45.017 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=deleted 2024-09-17T23:33:45.017 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:33:45.018 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:33:45.018 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:33:45.018 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_19 2024-09-17T23:33:45.018 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:33:45.018 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:33:45.018 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_19 2024-09-17T23:33:45.018 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_19 2024-09-17T23:33:45.018 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:33:45.050 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_19: (2) No such file or directory 2024-09-17T23:33:45.053 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image_id= 2024-09-17T23:33:45.053 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-17T23:33:45.053 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.1 2024-09-17T23:33:45.155 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_present cluster1 mirror image_19 deleted '' 2024-09-17T23:33:45.155 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:33:45.155 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:45.155 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_19 2024-09-17T23:33:45.155 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=deleted 2024-09-17T23:33:45.155 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:33:45.155 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=deleted 2024-09-17T23:33:45.155 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_image_id 2024-09-17T23:33:45.156 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_19 2024-09-17T23:33:45.156 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:33:45.156 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:33:45.156 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_19 2024-09-17T23:33:45.156 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_19 2024-09-17T23:33:45.156 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:33:45.189 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_19: (2) No such file or directory 2024-09-17T23:33:45.193 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_image_id= 2024-09-17T23:33:45.193 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:33:45.193 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test deleted = deleted 2024-09-17T23:33:45.193 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:33:45.193 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:33:45.193 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_20 2024-09-17T23:33:45.193 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_present cluster1 mirror image_20 deleted 2024-09-17T23:33:45.193 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:33:45.193 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:45.193 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_20 2024-09-17T23:33:45.193 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=deleted 2024-09-17T23:33:45.194 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:33:45.194 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:33:45.194 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:33:45.194 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_20 2024-09-17T23:33:45.194 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:33:45.194 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:33:45.194 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_20 2024-09-17T23:33:45.194 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_20 2024-09-17T23:33:45.194 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:33:45.225 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_20: (2) No such file or directory 2024-09-17T23:33:45.228 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image_id= 2024-09-17T23:33:45.228 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-17T23:33:45.229 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.1 2024-09-17T23:33:45.331 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_present cluster1 mirror image_20 deleted '' 2024-09-17T23:33:45.331 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:33:45.331 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:45.331 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_20 2024-09-17T23:33:45.331 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=deleted 2024-09-17T23:33:45.331 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:33:45.331 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=deleted 2024-09-17T23:33:45.331 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_image_id 2024-09-17T23:33:45.331 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_20 2024-09-17T23:33:45.332 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:33:45.332 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:33:45.332 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_20 2024-09-17T23:33:45.332 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_20 2024-09-17T23:33:45.332 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:33:45.370 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_20: (2) No such file or directory 2024-09-17T23:33:45.374 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_image_id= 2024-09-17T23:33:45.374 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:33:45.374 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test deleted = deleted 2024-09-17T23:33:45.375 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:33:45.375 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:33:45.375 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_21 2024-09-17T23:33:45.375 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_present cluster1 mirror image_21 deleted 2024-09-17T23:33:45.375 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:33:45.375 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:45.375 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_21 2024-09-17T23:33:45.375 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=deleted 2024-09-17T23:33:45.375 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:33:45.375 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:33:45.375 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:33:45.375 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_21 2024-09-17T23:33:45.375 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:33:45.376 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:33:45.376 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_21 2024-09-17T23:33:45.376 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_21 2024-09-17T23:33:45.376 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:33:45.415 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_21: (2) No such file or directory 2024-09-17T23:33:45.418 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image_id= 2024-09-17T23:33:45.418 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-17T23:33:45.418 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.1 2024-09-17T23:33:45.520 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_present cluster1 mirror image_21 deleted '' 2024-09-17T23:33:45.520 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:33:45.520 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:45.520 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_21 2024-09-17T23:33:45.520 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=deleted 2024-09-17T23:33:45.520 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:33:45.520 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=deleted 2024-09-17T23:33:45.520 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_image_id 2024-09-17T23:33:45.520 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_21 2024-09-17T23:33:45.521 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:33:45.521 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:33:45.521 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_21 2024-09-17T23:33:45.521 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_21 2024-09-17T23:33:45.521 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:33:45.553 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_21: (2) No such file or directory 2024-09-17T23:33:45.557 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_image_id= 2024-09-17T23:33:45.557 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:33:45.557 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test deleted = deleted 2024-09-17T23:33:45.557 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:33:45.557 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:33:45.558 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_22 2024-09-17T23:33:45.558 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_present cluster1 mirror image_22 deleted 2024-09-17T23:33:45.558 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:33:45.558 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:45.558 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_22 2024-09-17T23:33:45.558 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=deleted 2024-09-17T23:33:45.558 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:33:45.558 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:33:45.558 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:33:45.558 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_22 2024-09-17T23:33:45.558 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:33:45.558 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:33:45.558 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_22 2024-09-17T23:33:45.558 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_22 2024-09-17T23:33:45.558 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:33:45.590 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_22: (2) No such file or directory 2024-09-17T23:33:45.593 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image_id= 2024-09-17T23:33:45.593 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-17T23:33:45.593 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.1 2024-09-17T23:33:45.695 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_present cluster1 mirror image_22 deleted '' 2024-09-17T23:33:45.695 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:33:45.695 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:45.695 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_22 2024-09-17T23:33:45.695 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=deleted 2024-09-17T23:33:45.695 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:33:45.695 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=deleted 2024-09-17T23:33:45.695 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_image_id 2024-09-17T23:33:45.695 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_22 2024-09-17T23:33:45.695 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:33:45.695 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:33:45.696 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_22 2024-09-17T23:33:45.696 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_22 2024-09-17T23:33:45.696 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:33:45.730 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_22: (2) No such file or directory 2024-09-17T23:33:45.734 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_image_id= 2024-09-17T23:33:45.734 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:33:45.734 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test deleted = deleted 2024-09-17T23:33:45.734 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:33:45.734 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:33:45.734 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_23 2024-09-17T23:33:45.734 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_present cluster1 mirror image_23 deleted 2024-09-17T23:33:45.735 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:33:45.735 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:45.735 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_23 2024-09-17T23:33:45.735 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=deleted 2024-09-17T23:33:45.735 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:33:45.735 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:33:45.735 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:33:45.735 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_23 2024-09-17T23:33:45.735 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:33:45.735 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:33:45.735 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_23 2024-09-17T23:33:45.735 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_23 2024-09-17T23:33:45.735 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:33:45.768 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_23: (2) No such file or directory 2024-09-17T23:33:45.770 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image_id= 2024-09-17T23:33:45.770 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-17T23:33:45.771 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.1 2024-09-17T23:33:45.872 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_present cluster1 mirror image_23 deleted '' 2024-09-17T23:33:45.872 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:33:45.872 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:45.872 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_23 2024-09-17T23:33:45.872 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=deleted 2024-09-17T23:33:45.872 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:33:45.873 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=deleted 2024-09-17T23:33:45.873 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_image_id 2024-09-17T23:33:45.873 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_23 2024-09-17T23:33:45.873 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:33:45.873 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:33:45.873 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_23 2024-09-17T23:33:45.873 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_23 2024-09-17T23:33:45.873 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:33:45.906 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_23: (2) No such file or directory 2024-09-17T23:33:45.910 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_image_id= 2024-09-17T23:33:45.910 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:33:45.911 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test deleted = deleted 2024-09-17T23:33:45.911 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:33:45.911 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:33:45.911 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_24 2024-09-17T23:33:45.911 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_present cluster1 mirror image_24 deleted 2024-09-17T23:33:45.911 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:33:45.911 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:45.911 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_24 2024-09-17T23:33:45.911 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=deleted 2024-09-17T23:33:45.911 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:33:45.911 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:33:45.911 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:33:45.911 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_24 2024-09-17T23:33:45.911 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:33:45.911 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:33:45.911 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_24 2024-09-17T23:33:45.912 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_24 2024-09-17T23:33:45.912 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:33:45.943 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_24: (2) No such file or directory 2024-09-17T23:33:45.946 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image_id= 2024-09-17T23:33:45.946 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-17T23:33:45.946 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.1 2024-09-17T23:33:46.048 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_present cluster1 mirror image_24 deleted '' 2024-09-17T23:33:46.048 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:33:46.048 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:46.048 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_24 2024-09-17T23:33:46.048 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=deleted 2024-09-17T23:33:46.048 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:33:46.048 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=deleted 2024-09-17T23:33:46.048 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_image_id 2024-09-17T23:33:46.048 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_24 2024-09-17T23:33:46.049 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:33:46.049 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:33:46.049 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_24 2024-09-17T23:33:46.049 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_24 2024-09-17T23:33:46.049 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:33:46.081 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_24: (2) No such file or directory 2024-09-17T23:33:46.086 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_image_id= 2024-09-17T23:33:46.086 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:33:46.086 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test deleted = deleted 2024-09-17T23:33:46.086 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:33:46.086 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:33:46.086 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_25 2024-09-17T23:33:46.086 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_present cluster1 mirror image_25 deleted 2024-09-17T23:33:46.086 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:33:46.086 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:46.086 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_25 2024-09-17T23:33:46.086 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=deleted 2024-09-17T23:33:46.086 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:33:46.086 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:33:46.086 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:33:46.087 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_25 2024-09-17T23:33:46.087 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:33:46.087 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:33:46.087 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_25 2024-09-17T23:33:46.087 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_25 2024-09-17T23:33:46.087 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:33:46.119 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_25: (2) No such file or directory 2024-09-17T23:33:46.122 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image_id= 2024-09-17T23:33:46.122 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-17T23:33:46.122 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.1 2024-09-17T23:33:46.224 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_present cluster1 mirror image_25 deleted '' 2024-09-17T23:33:46.224 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:33:46.224 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:46.224 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_25 2024-09-17T23:33:46.224 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=deleted 2024-09-17T23:33:46.224 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:33:46.224 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=deleted 2024-09-17T23:33:46.224 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_image_id 2024-09-17T23:33:46.224 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_25 2024-09-17T23:33:46.224 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:33:46.225 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:33:46.225 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_25 2024-09-17T23:33:46.225 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_25 2024-09-17T23:33:46.225 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:33:46.258 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_25: (2) No such file or directory 2024-09-17T23:33:46.263 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_image_id= 2024-09-17T23:33:46.263 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:33:46.263 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test deleted = deleted 2024-09-17T23:33:46.263 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:33:46.263 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:33:46.263 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_26 2024-09-17T23:33:46.263 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_present cluster1 mirror image_26 deleted 2024-09-17T23:33:46.263 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:33:46.263 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:46.263 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_26 2024-09-17T23:33:46.263 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=deleted 2024-09-17T23:33:46.263 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:33:46.263 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:33:46.264 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:33:46.264 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_26 2024-09-17T23:33:46.264 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:33:46.264 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:33:46.264 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_26 2024-09-17T23:33:46.264 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_26 2024-09-17T23:33:46.264 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:33:46.295 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_26: (2) No such file or directory 2024-09-17T23:33:46.299 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image_id= 2024-09-17T23:33:46.299 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-17T23:33:46.299 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.1 2024-09-17T23:33:46.401 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_present cluster1 mirror image_26 deleted '' 2024-09-17T23:33:46.401 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:33:46.401 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:46.401 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_26 2024-09-17T23:33:46.401 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=deleted 2024-09-17T23:33:46.401 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:33:46.402 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=deleted 2024-09-17T23:33:46.402 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_image_id 2024-09-17T23:33:46.402 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_26 2024-09-17T23:33:46.402 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:33:46.402 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:33:46.402 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_26 2024-09-17T23:33:46.402 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_26 2024-09-17T23:33:46.402 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:33:46.434 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_26: (2) No such file or directory 2024-09-17T23:33:46.438 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_image_id= 2024-09-17T23:33:46.438 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:33:46.438 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test deleted = deleted 2024-09-17T23:33:46.438 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:33:46.438 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:33:46.438 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_27 2024-09-17T23:33:46.438 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_present cluster1 mirror image_27 deleted 2024-09-17T23:33:46.438 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:33:46.438 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:46.439 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_27 2024-09-17T23:33:46.439 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=deleted 2024-09-17T23:33:46.439 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:33:46.439 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:33:46.439 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:33:46.439 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_27 2024-09-17T23:33:46.439 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:33:46.439 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:33:46.439 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_27 2024-09-17T23:33:46.439 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_27 2024-09-17T23:33:46.439 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:33:46.471 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_27: (2) No such file or directory 2024-09-17T23:33:46.475 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image_id= 2024-09-17T23:33:46.475 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-17T23:33:46.475 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.1 2024-09-17T23:33:46.577 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_present cluster1 mirror image_27 deleted '' 2024-09-17T23:33:46.577 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:33:46.577 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:46.577 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_27 2024-09-17T23:33:46.577 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=deleted 2024-09-17T23:33:46.577 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:33:46.577 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=deleted 2024-09-17T23:33:46.577 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_image_id 2024-09-17T23:33:46.577 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_27 2024-09-17T23:33:46.578 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:33:46.578 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:33:46.578 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_27 2024-09-17T23:33:46.578 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_27 2024-09-17T23:33:46.578 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:33:46.612 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_27: (2) No such file or directory 2024-09-17T23:33:46.616 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_image_id= 2024-09-17T23:33:46.616 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:33:46.616 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test deleted = deleted 2024-09-17T23:33:46.616 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:33:46.616 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:33:46.616 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_28 2024-09-17T23:33:46.616 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_present cluster1 mirror image_28 deleted 2024-09-17T23:33:46.616 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:33:46.617 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:46.617 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_28 2024-09-17T23:33:46.617 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=deleted 2024-09-17T23:33:46.617 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:33:46.617 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:33:46.617 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:33:46.617 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_28 2024-09-17T23:33:46.617 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:33:46.617 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:33:46.617 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_28 2024-09-17T23:33:46.617 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_28 2024-09-17T23:33:46.617 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:33:46.649 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_28: (2) No such file or directory 2024-09-17T23:33:46.652 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image_id= 2024-09-17T23:33:46.652 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-17T23:33:46.652 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.1 2024-09-17T23:33:46.754 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_present cluster1 mirror image_28 deleted '' 2024-09-17T23:33:46.754 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:33:46.754 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:46.755 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_28 2024-09-17T23:33:46.755 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=deleted 2024-09-17T23:33:46.755 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:33:46.755 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=deleted 2024-09-17T23:33:46.755 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_image_id 2024-09-17T23:33:46.755 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_28 2024-09-17T23:33:46.755 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:33:46.755 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:33:46.755 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_28 2024-09-17T23:33:46.755 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_28 2024-09-17T23:33:46.755 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:33:46.789 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_28: (2) No such file or directory 2024-09-17T23:33:46.794 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_image_id= 2024-09-17T23:33:46.794 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:33:46.794 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test deleted = deleted 2024-09-17T23:33:46.794 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:33:46.794 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:33:46.794 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_29 2024-09-17T23:33:46.794 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_present cluster1 mirror image_29 deleted 2024-09-17T23:33:46.794 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:33:46.794 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:46.794 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_29 2024-09-17T23:33:46.794 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=deleted 2024-09-17T23:33:46.794 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:33:46.794 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:33:46.795 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:33:46.795 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_29 2024-09-17T23:33:46.795 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:33:46.795 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:33:46.795 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_29 2024-09-17T23:33:46.795 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_29 2024-09-17T23:33:46.795 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:33:46.826 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_29: (2) No such file or directory 2024-09-17T23:33:46.829 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image_id= 2024-09-17T23:33:46.829 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-17T23:33:46.829 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.1 2024-09-17T23:33:46.931 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_present cluster1 mirror image_29 deleted '' 2024-09-17T23:33:46.932 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:33:46.932 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:46.932 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_29 2024-09-17T23:33:46.932 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=deleted 2024-09-17T23:33:46.932 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:33:46.932 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=deleted 2024-09-17T23:33:46.932 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_image_id 2024-09-17T23:33:46.932 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_29 2024-09-17T23:33:46.932 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:33:46.932 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:33:46.932 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_29 2024-09-17T23:33:46.932 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_29 2024-09-17T23:33:46.932 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:33:46.965 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_29: (2) No such file or directory 2024-09-17T23:33:46.969 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_image_id= 2024-09-17T23:33:46.969 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:33:46.969 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test deleted = deleted 2024-09-17T23:33:46.969 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:33:46.969 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:33:46.969 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_30 2024-09-17T23:33:46.969 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_present cluster1 mirror image_30 deleted 2024-09-17T23:33:46.970 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:33:46.970 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:46.970 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_30 2024-09-17T23:33:46.970 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=deleted 2024-09-17T23:33:46.970 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:33:46.970 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:33:46.970 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:33:46.970 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_30 2024-09-17T23:33:46.970 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:33:46.970 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:33:46.970 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_30 2024-09-17T23:33:46.970 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_30 2024-09-17T23:33:46.970 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:33:47.002 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_30: (2) No such file or directory 2024-09-17T23:33:47.005 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image_id= 2024-09-17T23:33:47.006 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-17T23:33:47.006 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.1 2024-09-17T23:33:47.107 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_present cluster1 mirror image_30 deleted '' 2024-09-17T23:33:47.107 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:33:47.108 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:47.108 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_30 2024-09-17T23:33:47.108 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=deleted 2024-09-17T23:33:47.108 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:33:47.108 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=deleted 2024-09-17T23:33:47.108 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_image_id 2024-09-17T23:33:47.108 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_30 2024-09-17T23:33:47.108 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:33:47.108 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:33:47.108 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_30 2024-09-17T23:33:47.108 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_30 2024-09-17T23:33:47.108 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:33:47.141 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_30: (2) No such file or directory 2024-09-17T23:33:47.145 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_image_id= 2024-09-17T23:33:47.145 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:33:47.145 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test deleted = deleted 2024-09-17T23:33:47.145 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:33:47.145 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:33:47.146 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_31 2024-09-17T23:33:47.146 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_present cluster1 mirror image_31 deleted 2024-09-17T23:33:47.146 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:33:47.146 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:47.146 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_31 2024-09-17T23:33:47.146 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=deleted 2024-09-17T23:33:47.146 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:33:47.146 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:33:47.146 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:33:47.146 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_31 2024-09-17T23:33:47.146 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:33:47.146 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:33:47.146 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_31 2024-09-17T23:33:47.147 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_31 2024-09-17T23:33:47.147 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:33:47.179 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_31: (2) No such file or directory 2024-09-17T23:33:47.181 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image_id= 2024-09-17T23:33:47.182 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-17T23:33:47.182 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.1 2024-09-17T23:33:47.284 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_present cluster1 mirror image_31 deleted '' 2024-09-17T23:33:47.284 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:33:47.284 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:47.284 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_31 2024-09-17T23:33:47.284 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=deleted 2024-09-17T23:33:47.284 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:33:47.284 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=deleted 2024-09-17T23:33:47.284 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_image_id 2024-09-17T23:33:47.284 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_31 2024-09-17T23:33:47.284 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:33:47.284 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:33:47.285 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_31 2024-09-17T23:33:47.285 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_31 2024-09-17T23:33:47.285 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:33:47.320 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_31: (2) No such file or directory 2024-09-17T23:33:47.323 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_image_id= 2024-09-17T23:33:47.324 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:33:47.324 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test deleted = deleted 2024-09-17T23:33:47.324 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:33:47.324 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:33:47.324 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_32 2024-09-17T23:33:47.324 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_present cluster1 mirror image_32 deleted 2024-09-17T23:33:47.324 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:33:47.324 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:47.324 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_32 2024-09-17T23:33:47.324 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=deleted 2024-09-17T23:33:47.324 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:33:47.324 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:33:47.324 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:33:47.324 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_32 2024-09-17T23:33:47.325 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:33:47.325 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:33:47.325 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_32 2024-09-17T23:33:47.325 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_32 2024-09-17T23:33:47.325 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:33:47.357 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_32: (2) No such file or directory 2024-09-17T23:33:47.360 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image_id= 2024-09-17T23:33:47.360 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-17T23:33:47.360 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.1 2024-09-17T23:33:47.462 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_present cluster1 mirror image_32 deleted '' 2024-09-17T23:33:47.462 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:33:47.462 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:47.462 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_32 2024-09-17T23:33:47.462 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=deleted 2024-09-17T23:33:47.462 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:33:47.462 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=deleted 2024-09-17T23:33:47.462 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_image_id 2024-09-17T23:33:47.462 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_32 2024-09-17T23:33:47.462 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:33:47.463 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:33:47.463 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_32 2024-09-17T23:33:47.463 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_32 2024-09-17T23:33:47.463 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:33:47.498 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_32: (2) No such file or directory 2024-09-17T23:33:47.502 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_image_id= 2024-09-17T23:33:47.502 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:33:47.502 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test deleted = deleted 2024-09-17T23:33:47.502 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:33:47.502 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:33:47.502 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_33 2024-09-17T23:33:47.502 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_present cluster1 mirror image_33 deleted 2024-09-17T23:33:47.503 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:33:47.503 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:47.503 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_33 2024-09-17T23:33:47.503 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=deleted 2024-09-17T23:33:47.503 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:33:47.503 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:33:47.503 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:33:47.503 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_33 2024-09-17T23:33:47.503 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:33:47.503 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:33:47.503 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_33 2024-09-17T23:33:47.503 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_33 2024-09-17T23:33:47.503 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:33:47.535 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_33: (2) No such file or directory 2024-09-17T23:33:47.538 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image_id= 2024-09-17T23:33:47.538 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-17T23:33:47.538 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.1 2024-09-17T23:33:47.639 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_present cluster1 mirror image_33 deleted '' 2024-09-17T23:33:47.640 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:33:47.640 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:47.640 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_33 2024-09-17T23:33:47.640 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=deleted 2024-09-17T23:33:47.640 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:33:47.640 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=deleted 2024-09-17T23:33:47.640 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_image_id 2024-09-17T23:33:47.640 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_33 2024-09-17T23:33:47.640 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:33:47.640 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:33:47.640 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_33 2024-09-17T23:33:47.641 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_33 2024-09-17T23:33:47.641 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:33:47.675 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_33: (2) No such file or directory 2024-09-17T23:33:47.679 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_image_id= 2024-09-17T23:33:47.679 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:33:47.679 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test deleted = deleted 2024-09-17T23:33:47.680 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:33:47.680 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:33:47.680 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_34 2024-09-17T23:33:47.680 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_present cluster1 mirror image_34 deleted 2024-09-17T23:33:47.680 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:33:47.680 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:47.680 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_34 2024-09-17T23:33:47.680 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=deleted 2024-09-17T23:33:47.680 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:33:47.680 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:33:47.680 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:33:47.680 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_34 2024-09-17T23:33:47.680 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:33:47.681 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:33:47.681 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_34 2024-09-17T23:33:47.681 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_34 2024-09-17T23:33:47.681 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:33:47.713 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_34: (2) No such file or directory 2024-09-17T23:33:47.716 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image_id= 2024-09-17T23:33:47.716 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-17T23:33:47.716 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.1 2024-09-17T23:33:47.818 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_present cluster1 mirror image_34 deleted '' 2024-09-17T23:33:47.818 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:33:47.818 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:47.818 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_34 2024-09-17T23:33:47.818 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=deleted 2024-09-17T23:33:47.819 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:33:47.819 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=deleted 2024-09-17T23:33:47.819 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_image_id 2024-09-17T23:33:47.819 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_34 2024-09-17T23:33:47.819 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:33:47.819 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:33:47.819 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_34 2024-09-17T23:33:47.819 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_34 2024-09-17T23:33:47.819 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:33:47.852 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_34: (2) No such file or directory 2024-09-17T23:33:47.856 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_image_id= 2024-09-17T23:33:47.856 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:33:47.856 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test deleted = deleted 2024-09-17T23:33:47.857 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:33:47.857 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:33:47.857 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_35 2024-09-17T23:33:47.857 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_present cluster1 mirror image_35 deleted 2024-09-17T23:33:47.857 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:33:47.857 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:47.857 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_35 2024-09-17T23:33:47.857 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=deleted 2024-09-17T23:33:47.857 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:33:47.857 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:33:47.857 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:33:47.857 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_35 2024-09-17T23:33:47.858 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:33:47.858 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:33:47.858 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_35 2024-09-17T23:33:47.858 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_35 2024-09-17T23:33:47.858 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:33:48.091 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_35: (2) No such file or directory 2024-09-17T23:33:48.094 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image_id= 2024-09-17T23:33:48.094 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-17T23:33:48.095 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.1 2024-09-17T23:33:48.196 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_present cluster1 mirror image_35 deleted '' 2024-09-17T23:33:48.196 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:33:48.197 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:48.197 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_35 2024-09-17T23:33:48.197 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=deleted 2024-09-17T23:33:48.197 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:33:48.197 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=deleted 2024-09-17T23:33:48.197 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_image_id 2024-09-17T23:33:48.197 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_35 2024-09-17T23:33:48.197 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:33:48.197 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:33:48.197 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_35 2024-09-17T23:33:48.197 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_35 2024-09-17T23:33:48.197 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:33:48.233 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_35: (2) No such file or directory 2024-09-17T23:33:48.238 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_image_id= 2024-09-17T23:33:48.238 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:33:48.238 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test deleted = deleted 2024-09-17T23:33:48.238 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:33:48.238 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:33:48.238 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_36 2024-09-17T23:33:48.238 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_present cluster1 mirror image_36 deleted 2024-09-17T23:33:48.238 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:33:48.238 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:48.238 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_36 2024-09-17T23:33:48.238 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=deleted 2024-09-17T23:33:48.238 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:33:48.239 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:33:48.239 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:33:48.239 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_36 2024-09-17T23:33:48.239 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:33:48.239 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:33:48.239 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_36 2024-09-17T23:33:48.239 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_36 2024-09-17T23:33:48.239 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:33:48.270 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_36: (2) No such file or directory 2024-09-17T23:33:48.273 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image_id= 2024-09-17T23:33:48.273 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-17T23:33:48.274 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.1 2024-09-17T23:33:48.377 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_present cluster1 mirror image_36 deleted '' 2024-09-17T23:33:48.377 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:33:48.377 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:48.377 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_36 2024-09-17T23:33:48.377 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=deleted 2024-09-17T23:33:48.378 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:33:48.378 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=deleted 2024-09-17T23:33:48.378 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_image_id 2024-09-17T23:33:48.378 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_36 2024-09-17T23:33:48.378 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:33:48.378 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:33:48.379 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_36 2024-09-17T23:33:48.379 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_36 2024-09-17T23:33:48.379 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:33:48.410 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_36: (2) No such file or directory 2024-09-17T23:33:48.416 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_image_id= 2024-09-17T23:33:48.416 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:33:48.416 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test deleted = deleted 2024-09-17T23:33:48.416 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:33:48.417 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:33:48.417 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_37 2024-09-17T23:33:48.417 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_present cluster1 mirror image_37 deleted 2024-09-17T23:33:48.417 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:33:48.417 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:48.417 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_37 2024-09-17T23:33:48.417 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=deleted 2024-09-17T23:33:48.418 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:33:48.418 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:33:48.418 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:33:48.418 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_37 2024-09-17T23:33:48.418 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:33:48.418 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:33:48.418 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_37 2024-09-17T23:33:48.418 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_37 2024-09-17T23:33:48.418 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:33:48.448 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_37: (2) No such file or directory 2024-09-17T23:33:48.451 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image_id= 2024-09-17T23:33:48.451 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-17T23:33:48.451 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.1 2024-09-17T23:33:48.553 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_present cluster1 mirror image_37 deleted '' 2024-09-17T23:33:48.553 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:33:48.553 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:48.553 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_37 2024-09-17T23:33:48.553 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=deleted 2024-09-17T23:33:48.553 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:33:48.553 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=deleted 2024-09-17T23:33:48.553 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_image_id 2024-09-17T23:33:48.553 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_37 2024-09-17T23:33:48.554 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:33:48.554 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:33:48.554 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_37 2024-09-17T23:33:48.554 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_37 2024-09-17T23:33:48.554 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:33:48.588 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_37: (2) No such file or directory 2024-09-17T23:33:48.592 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_image_id= 2024-09-17T23:33:48.593 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:33:48.593 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test deleted = deleted 2024-09-17T23:33:48.593 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:33:48.593 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:33:48.593 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_38 2024-09-17T23:33:48.593 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_present cluster1 mirror image_38 deleted 2024-09-17T23:33:48.593 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:33:48.593 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:48.593 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_38 2024-09-17T23:33:48.593 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=deleted 2024-09-17T23:33:48.593 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:33:48.593 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:33:48.593 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:33:48.593 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_38 2024-09-17T23:33:48.594 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:33:48.594 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:33:48.594 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_38 2024-09-17T23:33:48.594 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_38 2024-09-17T23:33:48.594 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:33:48.625 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_38: (2) No such file or directory 2024-09-17T23:33:48.628 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image_id= 2024-09-17T23:33:48.628 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-17T23:33:48.629 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.1 2024-09-17T23:33:48.730 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_present cluster1 mirror image_38 deleted '' 2024-09-17T23:33:48.730 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:33:48.730 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:48.730 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_38 2024-09-17T23:33:48.730 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=deleted 2024-09-17T23:33:48.730 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:33:48.730 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=deleted 2024-09-17T23:33:48.730 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_image_id 2024-09-17T23:33:48.730 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_38 2024-09-17T23:33:48.731 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:33:48.731 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:33:48.731 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_38 2024-09-17T23:33:48.731 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_38 2024-09-17T23:33:48.731 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:33:48.764 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_38: (2) No such file or directory 2024-09-17T23:33:48.768 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_image_id= 2024-09-17T23:33:48.768 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:33:48.768 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test deleted = deleted 2024-09-17T23:33:48.768 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:33:48.768 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:33:48.768 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_39 2024-09-17T23:33:48.768 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_present cluster1 mirror image_39 deleted 2024-09-17T23:33:48.768 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:33:48.769 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:48.769 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_39 2024-09-17T23:33:48.769 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=deleted 2024-09-17T23:33:48.769 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:33:48.769 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:33:48.769 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:33:48.769 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_39 2024-09-17T23:33:48.769 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:33:48.769 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:33:48.769 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_39 2024-09-17T23:33:48.769 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_39 2024-09-17T23:33:48.769 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:33:48.801 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_39: (2) No such file or directory 2024-09-17T23:33:48.804 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image_id= 2024-09-17T23:33:48.804 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-17T23:33:48.805 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.1 2024-09-17T23:33:48.906 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_present cluster1 mirror image_39 deleted '' 2024-09-17T23:33:48.906 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:33:48.906 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:48.907 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_39 2024-09-17T23:33:48.907 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=deleted 2024-09-17T23:33:48.907 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:33:48.907 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=deleted 2024-09-17T23:33:48.907 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_image_id 2024-09-17T23:33:48.907 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_39 2024-09-17T23:33:48.907 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:33:48.907 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:33:48.907 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_39 2024-09-17T23:33:48.907 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_39 2024-09-17T23:33:48.907 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:33:48.940 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_39: (2) No such file or directory 2024-09-17T23:33:48.944 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_image_id= 2024-09-17T23:33:48.945 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:33:48.945 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test deleted = deleted 2024-09-17T23:33:48.945 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:33:48.945 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:33:48.945 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_40 2024-09-17T23:33:48.945 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_present cluster1 mirror image_40 deleted 2024-09-17T23:33:48.945 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:33:48.945 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:48.945 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_40 2024-09-17T23:33:48.945 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=deleted 2024-09-17T23:33:48.945 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:33:48.945 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:33:48.945 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:33:48.946 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_40 2024-09-17T23:33:48.946 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:33:48.946 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:33:48.946 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_40 2024-09-17T23:33:48.946 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_40 2024-09-17T23:33:48.946 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:33:48.978 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_40: (2) No such file or directory 2024-09-17T23:33:48.981 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image_id= 2024-09-17T23:33:48.982 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-17T23:33:48.982 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.1 2024-09-17T23:33:49.083 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_present cluster1 mirror image_40 deleted '' 2024-09-17T23:33:49.084 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:33:49.084 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:49.084 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_40 2024-09-17T23:33:49.084 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=deleted 2024-09-17T23:33:49.084 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:33:49.084 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=deleted 2024-09-17T23:33:49.084 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_image_id 2024-09-17T23:33:49.084 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_40 2024-09-17T23:33:49.084 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:33:49.084 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:33:49.084 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_40 2024-09-17T23:33:49.084 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_40 2024-09-17T23:33:49.084 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:33:49.118 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_40: (2) No such file or directory 2024-09-17T23:33:49.122 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_image_id= 2024-09-17T23:33:49.122 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:33:49.122 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test deleted = deleted 2024-09-17T23:33:49.122 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:33:49.122 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:33:49.122 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_41 2024-09-17T23:33:49.122 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_present cluster1 mirror image_41 deleted 2024-09-17T23:33:49.122 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:33:49.122 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:49.122 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_41 2024-09-17T23:33:49.122 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=deleted 2024-09-17T23:33:49.122 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:33:49.123 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:33:49.123 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:33:49.123 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_41 2024-09-17T23:33:49.123 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:33:49.123 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:33:49.123 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_41 2024-09-17T23:33:49.123 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_41 2024-09-17T23:33:49.123 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:33:49.156 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_41: (2) No such file or directory 2024-09-17T23:33:49.159 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image_id= 2024-09-17T23:33:49.159 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-17T23:33:49.159 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.1 2024-09-17T23:33:49.261 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_present cluster1 mirror image_41 deleted '' 2024-09-17T23:33:49.261 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:33:49.261 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:49.261 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_41 2024-09-17T23:33:49.261 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=deleted 2024-09-17T23:33:49.261 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:33:49.261 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=deleted 2024-09-17T23:33:49.262 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_image_id 2024-09-17T23:33:49.262 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_41 2024-09-17T23:33:49.262 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:33:49.262 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:33:49.262 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_41 2024-09-17T23:33:49.262 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_41 2024-09-17T23:33:49.262 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:33:49.296 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_41: (2) No such file or directory 2024-09-17T23:33:49.300 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_image_id= 2024-09-17T23:33:49.300 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:33:49.300 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test deleted = deleted 2024-09-17T23:33:49.301 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:33:49.301 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:33:49.301 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_42 2024-09-17T23:33:49.301 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_present cluster1 mirror image_42 deleted 2024-09-17T23:33:49.301 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:33:49.301 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:49.301 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_42 2024-09-17T23:33:49.301 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=deleted 2024-09-17T23:33:49.301 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:33:49.301 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:33:49.301 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:33:49.301 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_42 2024-09-17T23:33:49.301 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:33:49.302 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:33:49.302 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_42 2024-09-17T23:33:49.302 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_42 2024-09-17T23:33:49.302 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:33:49.333 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_42: (2) No such file or directory 2024-09-17T23:33:49.336 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image_id= 2024-09-17T23:33:49.336 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-17T23:33:49.336 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.1 2024-09-17T23:33:49.438 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_present cluster1 mirror image_42 deleted '' 2024-09-17T23:33:49.438 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:33:49.438 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:49.438 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_42 2024-09-17T23:33:49.438 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=deleted 2024-09-17T23:33:49.438 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:33:49.438 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=deleted 2024-09-17T23:33:49.438 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_image_id 2024-09-17T23:33:49.438 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_42 2024-09-17T23:33:49.438 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:33:49.438 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:33:49.438 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_42 2024-09-17T23:33:49.439 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_42 2024-09-17T23:33:49.439 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:33:49.471 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_42: (2) No such file or directory 2024-09-17T23:33:49.475 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_image_id= 2024-09-17T23:33:49.475 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:33:49.475 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test deleted = deleted 2024-09-17T23:33:49.475 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:33:49.475 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:33:49.475 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_43 2024-09-17T23:33:49.475 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_present cluster1 mirror image_43 deleted 2024-09-17T23:33:49.475 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:33:49.475 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:49.475 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_43 2024-09-17T23:33:49.475 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=deleted 2024-09-17T23:33:49.475 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:33:49.476 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:33:49.476 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:33:49.476 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_43 2024-09-17T23:33:49.476 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:33:49.476 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:33:49.476 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_43 2024-09-17T23:33:49.476 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_43 2024-09-17T23:33:49.476 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:33:49.508 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_43: (2) No such file or directory 2024-09-17T23:33:49.511 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image_id= 2024-09-17T23:33:49.511 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-17T23:33:49.511 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.1 2024-09-17T23:33:49.613 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_present cluster1 mirror image_43 deleted '' 2024-09-17T23:33:49.613 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:33:49.613 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:49.613 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_43 2024-09-17T23:33:49.613 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=deleted 2024-09-17T23:33:49.613 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:33:49.613 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=deleted 2024-09-17T23:33:49.614 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_image_id 2024-09-17T23:33:49.614 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_43 2024-09-17T23:33:49.614 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:33:49.614 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:33:49.614 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_43 2024-09-17T23:33:49.614 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_43 2024-09-17T23:33:49.614 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:33:49.646 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_43: (2) No such file or directory 2024-09-17T23:33:49.650 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_image_id= 2024-09-17T23:33:49.650 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:33:49.650 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test deleted = deleted 2024-09-17T23:33:49.651 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:33:49.651 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:33:49.651 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_44 2024-09-17T23:33:49.651 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_present cluster1 mirror image_44 deleted 2024-09-17T23:33:49.651 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:33:49.651 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:49.651 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_44 2024-09-17T23:33:49.651 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=deleted 2024-09-17T23:33:49.651 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:33:49.651 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:33:49.651 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:33:49.651 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_44 2024-09-17T23:33:49.651 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:33:49.651 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:33:49.651 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_44 2024-09-17T23:33:49.652 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_44 2024-09-17T23:33:49.652 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:33:49.683 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_44: (2) No such file or directory 2024-09-17T23:33:49.686 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image_id= 2024-09-17T23:33:49.686 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-17T23:33:49.686 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.1 2024-09-17T23:33:49.788 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_present cluster1 mirror image_44 deleted '' 2024-09-17T23:33:49.788 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:33:49.788 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:49.788 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_44 2024-09-17T23:33:49.788 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=deleted 2024-09-17T23:33:49.788 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:33:49.788 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=deleted 2024-09-17T23:33:49.789 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_image_id 2024-09-17T23:33:49.789 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_44 2024-09-17T23:33:49.789 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:33:49.789 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:33:49.789 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_44 2024-09-17T23:33:49.789 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_44 2024-09-17T23:33:49.789 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:33:49.821 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_44: (2) No such file or directory 2024-09-17T23:33:49.826 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_image_id= 2024-09-17T23:33:49.826 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:33:49.826 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test deleted = deleted 2024-09-17T23:33:49.826 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:33:49.826 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:33:49.826 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_45 2024-09-17T23:33:49.826 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_present cluster1 mirror image_45 deleted 2024-09-17T23:33:49.826 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:33:49.826 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:49.826 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_45 2024-09-17T23:33:49.826 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=deleted 2024-09-17T23:33:49.827 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:33:49.827 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:33:49.827 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:33:49.827 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_45 2024-09-17T23:33:49.827 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:33:49.827 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:33:49.827 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_45 2024-09-17T23:33:49.827 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_45 2024-09-17T23:33:49.827 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:33:49.860 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_45: (2) No such file or directory 2024-09-17T23:33:49.862 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image_id= 2024-09-17T23:33:49.863 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-17T23:33:49.863 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.1 2024-09-17T23:33:49.964 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_present cluster1 mirror image_45 deleted '' 2024-09-17T23:33:49.964 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:33:49.965 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:49.965 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_45 2024-09-17T23:33:49.965 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=deleted 2024-09-17T23:33:49.965 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:33:49.965 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=deleted 2024-09-17T23:33:49.965 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_image_id 2024-09-17T23:33:49.965 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_45 2024-09-17T23:33:49.965 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:33:49.965 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:33:49.965 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_45 2024-09-17T23:33:49.965 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_45 2024-09-17T23:33:49.965 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:33:49.975 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.7877684540298245) > weight (0.25) 2024-09-17T23:33:49.975 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2024-09-17T23:33:49.975 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2024-09-17T23:33:49.976 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2024-09-17T23:33:49.976 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi038.stderr:2024-09-17T23:33:49.974+0000 7f56723b5640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 41698) UID: 1000 2024-09-17T23:33:50.006 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_45: (2) No such file or directory 2024-09-17T23:33:50.011 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_image_id= 2024-09-17T23:33:50.011 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:33:50.011 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test deleted = deleted 2024-09-17T23:33:50.011 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:33:50.011 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:33:50.011 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_46 2024-09-17T23:33:50.012 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_present cluster1 mirror image_46 deleted 2024-09-17T23:33:50.012 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:33:50.012 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:50.012 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_46 2024-09-17T23:33:50.012 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=deleted 2024-09-17T23:33:50.012 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:33:50.012 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:33:50.012 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:33:50.012 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_46 2024-09-17T23:33:50.012 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:33:50.012 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:33:50.012 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_46 2024-09-17T23:33:50.012 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_46 2024-09-17T23:33:50.012 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:33:50.045 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_46: (2) No such file or directory 2024-09-17T23:33:50.048 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image_id= 2024-09-17T23:33:50.048 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-17T23:33:50.048 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.1 2024-09-17T23:33:50.150 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_present cluster1 mirror image_46 deleted '' 2024-09-17T23:33:50.150 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:33:50.150 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:50.150 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_46 2024-09-17T23:33:50.150 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=deleted 2024-09-17T23:33:50.150 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:33:50.150 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=deleted 2024-09-17T23:33:50.150 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_image_id 2024-09-17T23:33:50.150 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_46 2024-09-17T23:33:50.151 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:33:50.151 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:33:50.151 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_46 2024-09-17T23:33:50.151 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_46 2024-09-17T23:33:50.151 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:33:50.184 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_46: (2) No such file or directory 2024-09-17T23:33:50.188 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_image_id= 2024-09-17T23:33:50.188 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:33:50.188 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test deleted = deleted 2024-09-17T23:33:50.188 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:33:50.189 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:33:50.189 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_47 2024-09-17T23:33:50.189 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_present cluster1 mirror image_47 deleted 2024-09-17T23:33:50.189 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:33:50.189 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:50.189 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_47 2024-09-17T23:33:50.189 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=deleted 2024-09-17T23:33:50.189 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:33:50.189 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:33:50.189 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:33:50.189 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_47 2024-09-17T23:33:50.189 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:33:50.189 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:33:50.189 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_47 2024-09-17T23:33:50.190 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_47 2024-09-17T23:33:50.190 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:33:50.221 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_47: (2) No such file or directory 2024-09-17T23:33:50.224 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image_id= 2024-09-17T23:33:50.225 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-17T23:33:50.225 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.1 2024-09-17T23:33:50.326 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_present cluster1 mirror image_47 deleted '' 2024-09-17T23:33:50.326 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:33:50.327 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:50.327 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_47 2024-09-17T23:33:50.327 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=deleted 2024-09-17T23:33:50.327 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:33:50.327 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=deleted 2024-09-17T23:33:50.327 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_image_id 2024-09-17T23:33:50.327 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_47 2024-09-17T23:33:50.327 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:33:50.327 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:33:50.327 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_47 2024-09-17T23:33:50.327 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_47 2024-09-17T23:33:50.327 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:33:50.366 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_47: (2) No such file or directory 2024-09-17T23:33:50.371 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_image_id= 2024-09-17T23:33:50.371 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:33:50.371 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test deleted = deleted 2024-09-17T23:33:50.371 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:33:50.371 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:33:50.371 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_48 2024-09-17T23:33:50.371 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_present cluster1 mirror image_48 deleted 2024-09-17T23:33:50.371 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:33:50.371 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:50.371 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_48 2024-09-17T23:33:50.371 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=deleted 2024-09-17T23:33:50.371 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:33:50.371 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:33:50.372 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:33:50.372 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_48 2024-09-17T23:33:50.372 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:33:50.372 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:33:50.372 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_48 2024-09-17T23:33:50.372 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_48 2024-09-17T23:33:50.372 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:33:50.404 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_48: (2) No such file or directory 2024-09-17T23:33:50.407 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image_id= 2024-09-17T23:33:50.407 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-17T23:33:50.407 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.1 2024-09-17T23:33:50.509 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_present cluster1 mirror image_48 deleted '' 2024-09-17T23:33:50.510 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:33:50.510 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:50.510 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_48 2024-09-17T23:33:50.510 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=deleted 2024-09-17T23:33:50.510 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:33:50.510 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=deleted 2024-09-17T23:33:50.510 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_image_id 2024-09-17T23:33:50.510 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_48 2024-09-17T23:33:50.510 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:33:50.510 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:33:50.510 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_48 2024-09-17T23:33:50.510 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_48 2024-09-17T23:33:50.510 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:33:50.543 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_48: (2) No such file or directory 2024-09-17T23:33:50.548 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_image_id= 2024-09-17T23:33:50.548 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:33:50.548 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test deleted = deleted 2024-09-17T23:33:50.548 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:33:50.548 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:33:50.548 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_49 2024-09-17T23:33:50.548 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_present cluster1 mirror image_49 deleted 2024-09-17T23:33:50.548 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:33:50.548 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:50.548 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_49 2024-09-17T23:33:50.549 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=deleted 2024-09-17T23:33:50.549 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:33:50.549 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:33:50.549 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:33:50.549 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_49 2024-09-17T23:33:50.549 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:33:50.549 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:33:50.549 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_49 2024-09-17T23:33:50.549 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_49 2024-09-17T23:33:50.549 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:33:50.581 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_49: (2) No such file or directory 2024-09-17T23:33:50.584 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image_id= 2024-09-17T23:33:50.584 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-17T23:33:50.584 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.1 2024-09-17T23:33:50.686 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_present cluster1 mirror image_49 deleted '' 2024-09-17T23:33:50.686 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:33:50.686 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:50.686 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_49 2024-09-17T23:33:50.686 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=deleted 2024-09-17T23:33:50.687 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:33:50.687 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=deleted 2024-09-17T23:33:50.687 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_image_id 2024-09-17T23:33:50.687 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_49 2024-09-17T23:33:50.687 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:33:50.687 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:33:50.687 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_49 2024-09-17T23:33:50.687 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_49 2024-09-17T23:33:50.687 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:33:50.719 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_49: (2) No such file or directory 2024-09-17T23:33:50.723 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_image_id= 2024-09-17T23:33:50.724 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:33:50.724 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test deleted = deleted 2024-09-17T23:33:50.724 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:33:50.724 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-17T23:33:50.724 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_50 2024-09-17T23:33:50.724 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_present cluster1 mirror image_50 deleted 2024-09-17T23:33:50.724 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:33:50.724 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:50.724 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_50 2024-09-17T23:33:50.724 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=deleted 2024-09-17T23:33:50.725 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:33:50.725 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:33:50.725 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:33:50.725 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_50 2024-09-17T23:33:50.725 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:33:50.725 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:33:50.725 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_50 2024-09-17T23:33:50.725 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_50 2024-09-17T23:33:50.725 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:33:50.761 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_50: (2) No such file or directory 2024-09-17T23:33:50.764 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image_id= 2024-09-17T23:33:50.764 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-17T23:33:50.765 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.1 2024-09-17T23:33:50.867 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_present cluster1 mirror image_50 deleted '' 2024-09-17T23:33:50.867 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:33:50.867 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:50.867 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_50 2024-09-17T23:33:50.867 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=deleted 2024-09-17T23:33:50.868 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:33:50.868 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=deleted 2024-09-17T23:33:50.868 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_image_id 2024-09-17T23:33:50.868 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_50 2024-09-17T23:33:50.868 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:33:50.868 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:33:50.868 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_50 2024-09-17T23:33:50.868 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_50 2024-09-17T23:33:50.868 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:33:50.899 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_50: (2) No such file or directory 2024-09-17T23:33:50.902 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_image_id= 2024-09-17T23:33:50.903 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:33:50.903 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test deleted = deleted 2024-09-17T23:33:50.903 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:33:50.903 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ testlog 'TEST: delete images during bootstrap' 2024-09-17T23:33:50.903 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ tee -a /tmp/tmp.s5MEFDgsej/rbd-mirror.test.log 2024-09-17T23:33:50.903 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ date '+%F %T' 2024-09-17T23:33:50.904 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ echo 2024-09-17 23:33:50 TEST: delete images during bootstrap 2024-09-17T23:33:50.904 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:2024-09-17 23:33:50 TEST: delete images during bootstrap 2024-09-17T23:33:50.904 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set_pool_mirror_mode cluster1 mirror image 2024-09-17T23:33:50.904 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:33:50.904 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:50.904 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=image 2024-09-17T23:33:50.904 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster1 mirror pool enable mirror image 2024-09-17T23:33:50.937 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd: mirroring is already configured for image mode 2024-09-17T23:33:50.940 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set_pool_mirror_mode cluster2 mirror image 2024-09-17T23:33:50.940 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:33:50.940 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:50.940 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=image 2024-09-17T23:33:50.940 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 mirror pool enable mirror image 2024-09-17T23:33:50.972 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd: mirroring is already configured for image mode 2024-09-17T23:33:50.977 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ start_mirror cluster1 2024-09-17T23:33:50.977 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:33:50.977 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local instance 2024-09-17T23:33:50.977 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set_cluster_instance cluster1 cluster instance 2024-09-17T23:33:50.977 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local val=cluster1 2024-09-17T23:33:50.977 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster_var_name=cluster 2024-09-17T23:33:50.977 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local instance_var_name=instance 2024-09-17T23:33:50.977 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ cluster=cluster1 2024-09-17T23:33:50.978 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ instance=cluster1 2024-09-17T23:33:50.978 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' cluster1 = cluster1 ']' 2024-09-17T23:33:50.978 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ instance=0 2024-09-17T23:33:50.978 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ eval cluster=cluster1 2024-09-17T23:33:50.978 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ cluster=cluster1 2024-09-17T23:33:50.978 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ eval instance=0 2024-09-17T23:33:50.978 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ instance=0 2024-09-17T23:33:50.978 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n 1 2024-09-17T23:33:50.978 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 2024-09-17T23:33:50.978 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=test 2024-09-17T23:33:50.978 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ seq 1 10 2024-09-17T23:33:50.978 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 10` 2024-09-17T23:33:50.978 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_1 2024-09-17T23:33:50.979 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_image cluster2 mirror image_1 512M 2024-09-17T23:33:50.979 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:33:50.979 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:33:50.979 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:50.979 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:33:50.979 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_1 2024-09-17T23:33:50.979 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:33:50.979 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local size=128 2024-09-17T23:33:50.979 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n 512M ']' 2024-09-17T23:33:50.979 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ size=512M 2024-09-17T23:33:50.979 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:33:50.979 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_1 2024-09-17T23:33:51.028 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ enable_mirror cluster2 mirror image_1 2024-09-17T23:33:51.028 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:33:51.028 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:51.028 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_1 2024-09-17T23:33:51.028 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=snapshot 2024-09-17T23:33:51.028 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_1 snapshot 2024-09-17T23:33:51.221 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Mirroring enabled 2024-09-17T23:33:51.231 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 info mirror/image_1 2024-09-17T23:33:51.268 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'image_1': 2024-09-17T23:33:51.269 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 512 MiB in 128 objects 2024-09-17T23:33:51.269 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:33:51.269 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 1 2024-09-17T23:33:51.269 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 1952b4c0c2eb 2024-09-17T23:33:51.269 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.1952b4c0c2eb 2024-09-17T23:33:51.269 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:33:51.269 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-17T23:33:51.269 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:33:51.269 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:33:51.269 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:33:51 2024 2024-09-17T23:33:51.269 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:33:51 2024 2024-09-17T23:33:51.269 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:33:51 2024 2024-09-17T23:33:51.269 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:33:51.270 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:33:51.270 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: 83832d83-6c7b-42ec-95f6-2495c7a18f75 2024-09-17T23:33:51.270 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: true 2024-09-17T23:33:51.272 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ stress_write_image cluster2 mirror image_1 2024-09-17T23:33:51.272 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:33:51.272 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:33:51.272 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_1 2024-09-17T23:33:51.272 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-17T23:33:51.274 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local duration=14 2024-09-17T23:33:51.274 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set +e 2024-09-17T23:33:51.274 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_1 --debug-rbd=20 --debug-journaler=20 2024-09-17T23:33:51.527 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:100 IOs, 22385664 bytes 2024-09-17T23:33:51.649 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:200 IOs, 33170432 bytes 2024-09-17T23:33:51.786 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:300 IOs, 88784896 bytes 2024-09-17T23:33:51.923 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:400 IOs, 137764864 bytes 2024-09-17T23:33:52.021 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:500 IOs, 150786048 bytes 2024-09-17T23:33:52.120 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:600 IOs, 177861632 bytes 2024-09-17T23:33:52.222 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:700 IOs, 206976000 bytes 2024-09-17T23:33:52.348 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:800 IOs, 261979136 bytes 2024-09-17T23:33:52.462 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:900 IOs, 298773504 bytes 2024-09-17T23:33:52.601 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1000 IOs, 386225152 bytes 2024-09-17T23:33:52.727 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1100 IOs, 421874688 bytes 2024-09-17T23:33:52.817 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1200 IOs, 432135168 bytes 2024-09-17T23:33:52.864 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.3996195536601276) > weight (0.25) 2024-09-17T23:33:52.864 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2024-09-17T23:33:52.864 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2024-09-17T23:33:52.864 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2024-09-17T23:33:52.867 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi038.stderr:2024-09-17T23:33:52.865+0000 7f1181768640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 62139) UID: 1000 2024-09-17T23:33:52.955 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1300 IOs, 496008192 bytes 2024-09-17T23:33:53.073 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1400 IOs, 536192000 bytes 2024-09-17T23:33:53.153 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1500 IOs, 536811520 bytes 2024-09-17T23:33:53.283 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1600 IOs, 585587712 bytes 2024-09-17T23:33:53.363 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1700 IOs, 586196992 bytes 2024-09-17T23:33:53.480 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1800 IOs, 628489216 bytes 2024-09-17T23:33:53.607 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1900 IOs, 686223360 bytes 2024-09-17T23:33:53.714 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2000 IOs, 714090496 bytes 2024-09-17T23:33:53.836 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2100 IOs, 758738944 bytes 2024-09-17T23:33:53.967 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2200 IOs, 840563712 bytes 2024-09-17T23:33:54.110 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2300 IOs, 892548096 bytes 2024-09-17T23:33:54.254 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2400 IOs, 962922496 bytes 2024-09-17T23:33:54.363 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2500 IOs, 991959040 bytes 2024-09-17T23:33:54.472 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2600 IOs, 1028274176 bytes 2024-09-17T23:33:54.586 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2700 IOs, 1066822656 bytes 2024-09-17T23:33:54.697 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2800 IOs, 1102283776 bytes 2024-09-17T23:33:54.809 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2900 IOs, 1135361024 bytes 2024-09-17T23:33:54.910 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3000 IOs, 1161160704 bytes 2024-09-17T23:33:54.977 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2024-09-17T23:33:54.977 INFO:teuthology.orchestra.run:waiting for 600 2024-09-17T23:33:54.977 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2024-09-17T23:33:54.977 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2024-09-17T23:33:54.978 DEBUG:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2024-09-17T23:33:54.983 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2024-09-17T23:33:54.983 INFO:tasks.rbd_mirror_thrash:waiting for 118 secs before thrashing 2024-09-17T23:33:55.040 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3100 IOs, 1211721728 bytes 2024-09-17T23:33:55.206 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3200 IOs, 1287000064 bytes 2024-09-17T23:33:55.360 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3300 IOs, 1359307776 bytes 2024-09-17T23:33:55.497 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3400 IOs, 1400941568 bytes 2024-09-17T23:33:55.597 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3500 IOs, 1420504064 bytes 2024-09-17T23:33:55.728 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3600 IOs, 1469750272 bytes 2024-09-17T23:33:55.809 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3700 IOs, 1470358528 bytes 2024-09-17T23:33:55.922 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3800 IOs, 1512528896 bytes 2024-09-17T23:33:56.032 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3900 IOs, 1539414016 bytes 2024-09-17T23:33:56.150 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4000 IOs, 1579373568 bytes 2024-09-17T23:33:56.263 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4100 IOs, 1611493376 bytes 2024-09-17T23:33:56.365 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4200 IOs, 1656322048 bytes 2024-09-17T23:33:56.505 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4300 IOs, 1704175616 bytes 2024-09-17T23:33:56.651 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4400 IOs, 1769507840 bytes 2024-09-17T23:33:56.771 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4500 IOs, 1816732672 bytes 2024-09-17T23:33:56.886 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4600 IOs, 1856406528 bytes 2024-09-17T23:33:57.015 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4700 IOs, 1904267264 bytes 2024-09-17T23:33:57.111 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4800 IOs, 1942905856 bytes 2024-09-17T23:33:57.294 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4900 IOs, 2030077952 bytes 2024-09-17T23:33:57.444 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5000 IOs, 2088516608 bytes 2024-09-17T23:33:57.585 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5100 IOs, 2150762496 bytes 2024-09-17T23:33:57.705 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5200 IOs, 2192107520 bytes 2024-09-17T23:33:57.826 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5300 IOs, 2233740288 bytes 2024-09-17T23:33:57.865 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2024-09-17T23:33:57.865 INFO:teuthology.orchestra.run:waiting for 600 2024-09-17T23:33:57.865 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2024-09-17T23:33:57.866 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2024-09-17T23:33:57.866 DEBUG:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2024-09-17T23:33:57.869 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2024-09-17T23:33:57.869 INFO:tasks.rbd_mirror_thrash:waiting for 82 secs before thrashing 2024-09-17T23:33:57.956 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5400 IOs, 2261142528 bytes 2024-09-17T23:33:58.086 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5500 IOs, 2304385024 bytes 2024-09-17T23:33:58.247 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5600 IOs, 2382173184 bytes 2024-09-17T23:33:58.356 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5700 IOs, 2412381184 bytes 2024-09-17T23:33:58.487 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5800 IOs, 2461933568 bytes 2024-09-17T23:33:58.601 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5900 IOs, 2497664000 bytes 2024-09-17T23:33:58.746 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6000 IOs, 2569203712 bytes 2024-09-17T23:33:58.868 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6100 IOs, 2612728832 bytes 2024-09-17T23:33:58.972 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6200 IOs, 2641501184 bytes 2024-09-17T23:33:59.185 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6300 IOs, 2752510976 bytes 2024-09-17T23:33:59.335 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6400 IOs, 2828440576 bytes 2024-09-17T23:33:59.470 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6500 IOs, 2913054720 bytes 2024-09-17T23:33:59.591 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6600 IOs, 2951539712 bytes 2024-09-17T23:33:59.725 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6700 IOs, 3006184448 bytes 2024-09-17T23:33:59.855 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6800 IOs, 3060282368 bytes 2024-09-17T23:34:00.005 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6900 IOs, 3131502592 bytes 2024-09-17T23:34:00.100 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7000 IOs, 3155332096 bytes 2024-09-17T23:34:00.253 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7100 IOs, 3220594688 bytes 2024-09-17T23:34:00.365 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7200 IOs, 3254772736 bytes 2024-09-17T23:34:00.479 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7300 IOs, 3285687296 bytes 2024-09-17T23:34:00.568 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7400 IOs, 3289618432 bytes 2024-09-17T23:34:00.689 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7500 IOs, 3330813952 bytes 2024-09-17T23:34:00.801 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7600 IOs, 3364769792 bytes 2024-09-17T23:34:00.958 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7700 IOs, 3437201408 bytes 2024-09-17T23:34:01.096 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7800 IOs, 3492132864 bytes 2024-09-17T23:34:01.259 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7900 IOs, 3569526784 bytes 2024-09-17T23:34:01.367 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8000 IOs, 3599305728 bytes 2024-09-17T23:34:01.490 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8100 IOs, 3644207104 bytes 2024-09-17T23:34:01.604 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8200 IOs, 3672172544 bytes 2024-09-17T23:34:01.703 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8300 IOs, 3692737536 bytes 2024-09-17T23:34:01.817 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8400 IOs, 3727738880 bytes 2024-09-17T23:34:01.944 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8500 IOs, 3792284672 bytes 2024-09-17T23:34:02.082 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8600 IOs, 3818343424 bytes 2024-09-17T23:34:02.257 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8700 IOs, 3909419008 bytes 2024-09-17T23:34:02.387 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8800 IOs, 3962362880 bytes 2024-09-17T23:34:02.484 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8900 IOs, 3975697408 bytes 2024-09-17T23:34:02.610 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9000 IOs, 4025914368 bytes 2024-09-17T23:34:02.709 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9100 IOs, 4041674752 bytes 2024-09-17T23:34:02.830 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9200 IOs, 4085891072 bytes 2024-09-17T23:34:02.981 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9300 IOs, 4147621888 bytes 2024-09-17T23:34:03.113 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9400 IOs, 4200506368 bytes 2024-09-17T23:34:03.264 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9500 IOs, 4271159296 bytes 2024-09-17T23:34:03.436 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9600 IOs, 4369352704 bytes 2024-09-17T23:34:03.591 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9700 IOs, 4443176960 bytes 2024-09-17T23:34:03.697 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9800 IOs, 4466897920 bytes 2024-09-17T23:34:03.851 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9900 IOs, 4539796480 bytes 2024-09-17T23:34:03.975 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10000 IOs, 4581211136 bytes 2024-09-17T23:34:04.064 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10100 IOs, 4589602816 bytes 2024-09-17T23:34:04.165 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10200 IOs, 4606311424 bytes 2024-09-17T23:34:04.292 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10300 IOs, 4646530048 bytes 2024-09-17T23:34:04.396 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10400 IOs, 4669930496 bytes 2024-09-17T23:34:04.477 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10500 IOs, 4670528512 bytes 2024-09-17T23:34:04.594 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10600 IOs, 4704382976 bytes 2024-09-17T23:34:04.689 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10700 IOs, 4722049024 bytes 2024-09-17T23:34:04.857 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10800 IOs, 4810876928 bytes 2024-09-17T23:34:04.998 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10900 IOs, 4868965376 bytes 2024-09-17T23:34:05.129 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:11000 IOs, 4917606400 bytes 2024-09-17T23:34:05.233 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:11100 IOs, 4938575872 bytes 2024-09-17T23:34:05.282 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ error_code=124 2024-09-17T23:34:05.282 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set -e 2024-09-17T23:34:05.282 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' 124 -eq 124 ']' 2024-09-17T23:34:05.282 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:34:05.282 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_present cluster1 mirror image_1 present 2024-09-17T23:34:05.282 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:34:05.282 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:34:05.282 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_1 2024-09-17T23:34:05.282 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=present 2024-09-17T23:34:05.282 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:34:05.283 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:34:05.283 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:34:05.283 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_1 2024-09-17T23:34:05.283 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:34:05.284 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:34:05.284 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_1 2024-09-17T23:34:05.284 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-09-17T23:34:05.284 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:34:05.329 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image_id=12b57c3fe7bf 2024-09-17T23:34:05.329 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-17T23:34:05.329 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.1 2024-09-17T23:34:05.430 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_present cluster1 mirror image_1 present 12b57c3fe7bf 2024-09-17T23:34:05.431 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:34:05.431 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:34:05.431 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_1 2024-09-17T23:34:05.431 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=present 2024-09-17T23:34:05.431 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id=12b57c3fe7bf 2024-09-17T23:34:05.431 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=deleted 2024-09-17T23:34:05.431 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_image_id 2024-09-17T23:34:05.431 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_1 2024-09-17T23:34:05.431 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:34:05.431 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:34:05.431 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_1 2024-09-17T23:34:05.431 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-09-17T23:34:05.431 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:34:05.474 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_image_id=12b57c3fe7bf 2024-09-17T23:34:05.474 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n 12b57c3fe7bf 2024-09-17T23:34:05.474 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -z 12b57c3fe7bf -o 12b57c3fe7bf = 12b57c3fe7bf 2024-09-17T23:34:05.474 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_state=present 2024-09-17T23:34:05.474 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test present = present 2024-09-17T23:34:05.474 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:34:05.475 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ disable_mirror cluster2 mirror image_1 2024-09-17T23:34:05.475 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:34:05.475 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:34:05.475 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_1 2024-09-17T23:34:05.475 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_1 2024-09-17T23:34:26.916 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:34:26.914+0000 7ff50242c640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55d142d0f380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-17T23:34:26.917 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:34:26.914+0000 7ff508c39640 -1 rbd::mirror::ImageReplayer: 0x55d142730500 [3/83832d83-6c7b-42ec-95f6-2495c7a18f75] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-17T23:34:36.322 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi038.stderr:2024-09-17T23:34:36.320+0000 7efed93cd640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5562643fdc00 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=185 2024-09-17T23:34:36.323 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi038.stderr:2024-09-17T23:34:36.320+0000 7efed93cd640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5562643fdc00 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=185, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=0ba5654d-47fb-4672-b20c-7359a1340314, primary_snap_id=b9, last_copied_object_number=128, snap_seqs={185=18446744073709551614}] 2024-09-17T23:34:36.333 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Mirroring disabled 2024-09-17T23:34:36.355 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_present cluster1 mirror image_1 deleted 2024-09-17T23:34:36.355 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:34:36.355 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:34:36.355 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_1 2024-09-17T23:34:36.355 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=deleted 2024-09-17T23:34:36.355 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:34:36.355 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:34:36.355 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:34:36.356 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_1 2024-09-17T23:34:36.356 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:34:36.356 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:34:36.356 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_1 2024-09-17T23:34:36.356 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-09-17T23:34:36.356 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:34:36.396 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_1: (2) No such file or directory 2024-09-17T23:34:36.399 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image_id= 2024-09-17T23:34:36.399 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-17T23:34:36.399 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.1 2024-09-17T23:34:36.501 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_present cluster1 mirror image_1 deleted '' 2024-09-17T23:34:36.501 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:34:36.501 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:34:36.501 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_1 2024-09-17T23:34:36.501 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=deleted 2024-09-17T23:34:36.501 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:34:36.501 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=deleted 2024-09-17T23:34:36.502 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_image_id 2024-09-17T23:34:36.502 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_1 2024-09-17T23:34:36.502 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:34:36.502 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:34:36.502 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_1 2024-09-17T23:34:36.502 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-09-17T23:34:36.502 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:34:36.536 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_1: (2) No such file or directory 2024-09-17T23:34:36.540 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_image_id= 2024-09-17T23:34:36.540 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:34:36.540 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test deleted = deleted 2024-09-17T23:34:36.540 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:34:36.540 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ purge_snapshots cluster2 mirror image_1 2024-09-17T23:34:36.540 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:34:36.541 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:34:36.541 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_1 2024-09-17T23:34:36.541 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap purge mirror/image_1 2024-09-17T23:34:36.586 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image_retry cluster2 mirror image_1 2024-09-17T23:34:36.587 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:34:36.587 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:34:36.587 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_1 2024-09-17T23:34:36.587 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-17T23:34:36.587 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0 2024-09-17T23:34:36.588 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image cluster2 mirror image_1 2024-09-17T23:34:36.588 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:34:36.588 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:34:36.588 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_1 2024-09-17T23:34:36.588 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_1 2024-09-17T23:34:36.632 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 rm mirror/image_1 2024-09-17T23:34:36.889 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-17T23:34:36.893 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:34:36.893 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 10` 2024-09-17T23:34:36.893 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_2 2024-09-17T23:34:36.893 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_image cluster2 mirror image_2 512M 2024-09-17T23:34:36.893 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:34:36.893 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:34:36.893 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:34:36.894 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:34:36.894 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_2 2024-09-17T23:34:36.894 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:34:36.894 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local size=128 2024-09-17T23:34:36.894 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n 512M ']' 2024-09-17T23:34:36.894 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ size=512M 2024-09-17T23:34:36.894 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:34:36.894 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_2 2024-09-17T23:34:36.944 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ enable_mirror cluster2 mirror image_2 2024-09-17T23:34:36.944 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:34:36.944 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:34:36.944 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_2 2024-09-17T23:34:36.944 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=snapshot 2024-09-17T23:34:36.944 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_2 snapshot 2024-09-17T23:34:37.349 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Mirroring enabled 2024-09-17T23:34:37.361 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 info mirror/image_2 2024-09-17T23:34:37.600 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'image_2': 2024-09-17T23:34:37.601 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 512 MiB in 128 objects 2024-09-17T23:34:37.601 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:34:37.601 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 1 2024-09-17T23:34:37.601 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 1978d717d52e 2024-09-17T23:34:37.601 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.1978d717d52e 2024-09-17T23:34:37.601 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:34:37.601 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-17T23:34:37.601 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:34:37.601 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:34:37.601 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:34:36 2024 2024-09-17T23:34:37.601 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:34:36 2024 2024-09-17T23:34:37.601 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:34:36 2024 2024-09-17T23:34:37.601 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:34:37.602 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:34:37.602 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: 3a823554-e61b-4672-88e8-f9fd6f4e354d 2024-09-17T23:34:37.602 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: true 2024-09-17T23:34:37.604 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ stress_write_image cluster2 mirror image_2 2024-09-17T23:34:37.604 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:34:37.604 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:34:37.604 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_2 2024-09-17T23:34:37.604 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-17T23:34:37.606 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local duration=10 2024-09-17T23:34:37.606 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set +e 2024-09-17T23:34:37.606 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_2 --debug-rbd=20 --debug-journaler=20 2024-09-17T23:34:37.881 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:100 IOs, 60971008 bytes 2024-09-17T23:34:38.038 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:200 IOs, 114820096 bytes 2024-09-17T23:34:38.155 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:300 IOs, 137996288 bytes 2024-09-17T23:34:38.275 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:400 IOs, 180604928 bytes 2024-09-17T23:34:38.414 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:500 IOs, 238046208 bytes 2024-09-17T23:34:38.513 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:600 IOs, 256491520 bytes 2024-09-17T23:34:38.634 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:700 IOs, 307209216 bytes 2024-09-17T23:34:38.733 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:800 IOs, 327957504 bytes 2024-09-17T23:34:38.885 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:900 IOs, 415552512 bytes 2024-09-17T23:34:38.970 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1000 IOs, 416177152 bytes 2024-09-17T23:34:39.103 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1100 IOs, 475480064 bytes 2024-09-17T23:34:39.227 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1200 IOs, 523726848 bytes 2024-09-17T23:34:39.363 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1300 IOs, 584540160 bytes 2024-09-17T23:34:39.465 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1400 IOs, 605079552 bytes 2024-09-17T23:34:39.590 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1500 IOs, 665889792 bytes 2024-09-17T23:34:39.725 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1600 IOs, 723469312 bytes 2024-09-17T23:34:39.845 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1700 IOs, 765268992 bytes 2024-09-17T23:34:39.980 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1800 IOs, 835966976 bytes 2024-09-17T23:34:40.070 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1900 IOs, 836614144 bytes 2024-09-17T23:34:40.178 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2000 IOs, 861858816 bytes 2024-09-17T23:34:40.330 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2100 IOs, 958583808 bytes 2024-09-17T23:34:40.460 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2200 IOs, 997522432 bytes 2024-09-17T23:34:40.592 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2300 IOs, 1054692352 bytes 2024-09-17T23:34:40.735 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2400 IOs, 1124439040 bytes 2024-09-17T23:34:40.853 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2500 IOs, 1165622272 bytes 2024-09-17T23:34:40.977 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2600 IOs, 1212847104 bytes 2024-09-17T23:34:41.080 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2700 IOs, 1240379392 bytes 2024-09-17T23:34:41.212 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2800 IOs, 1298420736 bytes 2024-09-17T23:34:41.327 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2900 IOs, 1328899072 bytes 2024-09-17T23:34:41.416 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3000 IOs, 1335707648 bytes 2024-09-17T23:34:41.513 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3100 IOs, 1357686784 bytes 2024-09-17T23:34:41.662 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3200 IOs, 1431514112 bytes 2024-09-17T23:34:41.814 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3300 IOs, 1506877440 bytes 2024-09-17T23:34:41.954 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3400 IOs, 1570731008 bytes 2024-09-17T23:34:42.065 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3500 IOs, 1607260160 bytes 2024-09-17T23:34:42.177 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3600 IOs, 1632271360 bytes 2024-09-17T23:34:42.303 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3700 IOs, 1668842496 bytes 2024-09-17T23:34:42.390 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3800 IOs, 1673916416 bytes 2024-09-17T23:34:42.487 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3900 IOs, 1692576768 bytes 2024-09-17T23:34:42.609 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4000 IOs, 1734238208 bytes 2024-09-17T23:34:42.704 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4100 IOs, 1749177344 bytes 2024-09-17T23:34:42.730 INFO:tasks.ceph.cluster2.osd.2.smithi038.stderr:2024-09-17T23:34:42.728+0000 7f1c4b04a640 -1 reset not still connected to 0x561ce28f1860 2024-09-17T23:34:42.806 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4200 IOs, 1767881728 bytes 2024-09-17T23:34:42.902 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4300 IOs, 1784169472 bytes 2024-09-17T23:34:43.058 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4400 IOs, 1874027520 bytes 2024-09-17T23:34:43.228 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4500 IOs, 1951326208 bytes 2024-09-17T23:34:43.342 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4600 IOs, 1981244416 bytes 2024-09-17T23:34:43.459 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4700 IOs, 2024354816 bytes 2024-09-17T23:34:43.563 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4800 IOs, 2052187136 bytes 2024-09-17T23:34:43.663 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4900 IOs, 2076522496 bytes 2024-09-17T23:34:43.787 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5000 IOs, 2123234304 bytes 2024-09-17T23:34:43.899 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5100 IOs, 2159092736 bytes 2024-09-17T23:34:44.005 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5200 IOs, 2187719680 bytes 2024-09-17T23:34:44.164 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5300 IOs, 2266170368 bytes 2024-09-17T23:34:44.281 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5400 IOs, 2305406976 bytes 2024-09-17T23:34:44.375 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5500 IOs, 2319295488 bytes 2024-09-17T23:34:44.490 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5600 IOs, 2356477952 bytes 2024-09-17T23:34:44.678 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5700 IOs, 2474617856 bytes 2024-09-17T23:34:44.760 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5800 IOs, 2475243520 bytes 2024-09-17T23:34:44.873 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5900 IOs, 2515145728 bytes 2024-09-17T23:34:44.974 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6000 IOs, 2544198656 bytes 2024-09-17T23:34:45.112 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6100 IOs, 2592973824 bytes 2024-09-17T23:34:45.241 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6200 IOs, 2637175808 bytes 2024-09-17T23:34:45.357 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6300 IOs, 2677501952 bytes 2024-09-17T23:34:45.471 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6400 IOs, 2704240640 bytes 2024-09-17T23:34:45.556 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6500 IOs, 2710173696 bytes 2024-09-17T23:34:45.682 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6600 IOs, 2757661696 bytes 2024-09-17T23:34:45.791 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6700 IOs, 2794357760 bytes 2024-09-17T23:34:45.901 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6800 IOs, 2821774336 bytes 2024-09-17T23:34:46.015 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6900 IOs, 2856473600 bytes 2024-09-17T23:34:46.145 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7000 IOs, 2908065792 bytes 2024-09-17T23:34:46.313 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7100 IOs, 3000750080 bytes 2024-09-17T23:34:46.420 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7200 IOs, 3039088640 bytes 2024-09-17T23:34:46.556 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7300 IOs, 3083203584 bytes 2024-09-17T23:34:46.650 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7400 IOs, 3098442752 bytes 2024-09-17T23:34:46.738 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7500 IOs, 3104041984 bytes 2024-09-17T23:34:46.837 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7600 IOs, 3124816896 bytes 2024-09-17T23:34:46.959 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7700 IOs, 3170963456 bytes 2024-09-17T23:34:47.069 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7800 IOs, 3200744448 bytes 2024-09-17T23:34:47.202 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7900 IOs, 3253456896 bytes 2024-09-17T23:34:47.314 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8000 IOs, 3283514368 bytes 2024-09-17T23:34:47.481 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8100 IOs, 3375509504 bytes 2024-09-17T23:34:47.595 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8200 IOs, 3430576128 bytes 2024-09-17T23:34:47.613 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ error_code=124 2024-09-17T23:34:47.613 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set -e 2024-09-17T23:34:47.613 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' 124 -eq 124 ']' 2024-09-17T23:34:47.614 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:34:47.614 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_present cluster1 mirror image_2 present 2024-09-17T23:34:47.614 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:34:47.614 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:34:47.614 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_2 2024-09-17T23:34:47.614 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=present 2024-09-17T23:34:47.614 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:34:47.614 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:34:47.614 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:34:47.614 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_2 2024-09-17T23:34:47.614 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:34:47.614 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:34:47.614 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_2 2024-09-17T23:34:47.615 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-09-17T23:34:47.615 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:34:47.660 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image_id=12b5b029f5b7 2024-09-17T23:34:47.660 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-17T23:34:47.660 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.1 2024-09-17T23:34:47.762 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_present cluster1 mirror image_2 present 12b5b029f5b7 2024-09-17T23:34:47.762 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:34:47.762 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:34:47.762 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_2 2024-09-17T23:34:47.763 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=present 2024-09-17T23:34:47.763 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id=12b5b029f5b7 2024-09-17T23:34:47.763 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=deleted 2024-09-17T23:34:47.763 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_image_id 2024-09-17T23:34:47.763 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_2 2024-09-17T23:34:47.763 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:34:47.763 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:34:47.764 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_2 2024-09-17T23:34:47.764 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-09-17T23:34:47.764 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:34:47.808 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_image_id=12b5b029f5b7 2024-09-17T23:34:47.808 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n 12b5b029f5b7 2024-09-17T23:34:47.808 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -z 12b5b029f5b7 -o 12b5b029f5b7 = 12b5b029f5b7 2024-09-17T23:34:47.808 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_state=present 2024-09-17T23:34:47.808 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test present = present 2024-09-17T23:34:47.808 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:34:47.808 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ disable_mirror cluster2 mirror image_2 2024-09-17T23:34:47.809 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:34:47.809 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:34:47.809 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_2 2024-09-17T23:34:47.809 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_2 2024-09-17T23:34:56.915 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:34:56.913+0000 7ff50242c640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55d142d0fba0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-17T23:34:56.915 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:34:56.913+0000 7ff508c39640 -1 rbd::mirror::ImageReplayer: 0x55d1438cc280 [3/3a823554-e61b-4672-88e8-f9fd6f4e354d] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-17T23:35:18.292 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi038.stderr:2024-09-17T23:35:18.290+0000 7efed93cd640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5562643a2000 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=186 2024-09-17T23:35:18.293 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi038.stderr:2024-09-17T23:35:18.290+0000 7efed93cd640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5562643a2000 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=186, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=0ba5654d-47fb-4672-b20c-7359a1340314, primary_snap_id=ba, last_copied_object_number=128, snap_seqs={186=18446744073709551614}] 2024-09-17T23:35:18.302 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Mirroring disabled 2024-09-17T23:35:18.330 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_present cluster1 mirror image_2 deleted 2024-09-17T23:35:18.330 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:35:18.331 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:35:18.331 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_2 2024-09-17T23:35:18.331 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=deleted 2024-09-17T23:35:18.331 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:35:18.331 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:35:18.331 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:35:18.331 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_2 2024-09-17T23:35:18.331 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:35:18.331 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:35:18.331 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_2 2024-09-17T23:35:18.331 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-09-17T23:35:18.331 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:35:18.369 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_2: (2) No such file or directory 2024-09-17T23:35:18.372 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image_id= 2024-09-17T23:35:18.372 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-17T23:35:18.373 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.1 2024-09-17T23:35:18.474 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_present cluster1 mirror image_2 deleted '' 2024-09-17T23:35:18.475 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:35:18.475 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:35:18.475 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_2 2024-09-17T23:35:18.475 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=deleted 2024-09-17T23:35:18.475 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:35:18.475 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=deleted 2024-09-17T23:35:18.475 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_image_id 2024-09-17T23:35:18.475 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_2 2024-09-17T23:35:18.475 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:35:18.475 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:35:18.475 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_2 2024-09-17T23:35:18.475 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-09-17T23:35:18.475 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:35:18.509 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_2: (2) No such file or directory 2024-09-17T23:35:18.513 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_image_id= 2024-09-17T23:35:18.513 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:35:18.513 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test deleted = deleted 2024-09-17T23:35:18.513 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:35:18.513 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ purge_snapshots cluster2 mirror image_2 2024-09-17T23:35:18.513 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:35:18.513 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:35:18.513 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_2 2024-09-17T23:35:18.513 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap purge mirror/image_2 2024-09-17T23:35:18.558 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image_retry cluster2 mirror image_2 2024-09-17T23:35:18.558 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:35:18.558 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:35:18.558 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_2 2024-09-17T23:35:18.558 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-17T23:35:18.558 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0 2024-09-17T23:35:18.559 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image cluster2 mirror image_2 2024-09-17T23:35:18.559 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:35:18.559 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:35:18.559 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_2 2024-09-17T23:35:18.559 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_2 2024-09-17T23:35:18.604 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 rm mirror/image_2 2024-09-17T23:35:18.868 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-17T23:35:18.872 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:35:18.872 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 10` 2024-09-17T23:35:18.872 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_3 2024-09-17T23:35:18.872 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_image cluster2 mirror image_3 512M 2024-09-17T23:35:18.872 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:35:18.872 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:35:18.872 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:35:18.872 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:35:18.872 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_3 2024-09-17T23:35:18.872 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:35:18.872 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local size=128 2024-09-17T23:35:18.873 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n 512M ']' 2024-09-17T23:35:18.873 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ size=512M 2024-09-17T23:35:18.873 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:35:18.873 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_3 2024-09-17T23:35:18.922 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ enable_mirror cluster2 mirror image_3 2024-09-17T23:35:18.922 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:35:18.923 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:35:18.923 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_3 2024-09-17T23:35:18.923 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=snapshot 2024-09-17T23:35:18.923 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_3 snapshot 2024-09-17T23:35:19.327 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Mirroring enabled 2024-09-17T23:35:19.337 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 info mirror/image_3 2024-09-17T23:35:19.376 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'image_3': 2024-09-17T23:35:19.376 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 512 MiB in 128 objects 2024-09-17T23:35:19.376 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:35:19.376 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 1 2024-09-17T23:35:19.376 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 1990d643f3ea 2024-09-17T23:35:19.377 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.1990d643f3ea 2024-09-17T23:35:19.377 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:35:19.377 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-17T23:35:19.377 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:35:19.377 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:35:19.377 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:35:18 2024 2024-09-17T23:35:19.377 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:35:18 2024 2024-09-17T23:35:19.377 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:35:18 2024 2024-09-17T23:35:19.377 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:35:19.377 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:35:19.377 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: 64771169-8de5-40e2-99f6-6a2c02c69326 2024-09-17T23:35:19.377 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: true 2024-09-17T23:35:19.379 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ stress_write_image cluster2 mirror image_3 2024-09-17T23:35:19.379 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:35:19.380 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:35:19.380 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_3 2024-09-17T23:35:19.380 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-17T23:35:19.381 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local duration=12 2024-09-17T23:35:19.381 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set +e 2024-09-17T23:35:19.381 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_3 --debug-rbd=20 --debug-journaler=20 2024-09-17T23:35:19.666 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:100 IOs, 64668672 bytes 2024-09-17T23:35:19.821 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:200 IOs, 112084992 bytes 2024-09-17T23:35:19.870 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-09-17T23:35:19.871 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-09-17T23:35:19.871 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2024-09-17T23:35:19.873 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi038.stderr:2024-09-17T23:35:19.870+0000 7f133563a640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 58826) UID: 1000 2024-09-17T23:35:19.934 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:300 IOs, 143484928 bytes 2024-09-17T23:35:20.074 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:400 IOs, 203503616 bytes 2024-09-17T23:35:20.195 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:500 IOs, 249461760 bytes 2024-09-17T23:35:20.322 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:600 IOs, 305347584 bytes 2024-09-17T23:35:20.458 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:700 IOs, 360729600 bytes 2024-09-17T23:35:20.546 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:800 IOs, 383442944 bytes 2024-09-17T23:35:20.641 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:900 IOs, 388940800 bytes 2024-09-17T23:35:20.747 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1000 IOs, 420035584 bytes 2024-09-17T23:35:20.876 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1100 IOs, 466641920 bytes 2024-09-17T23:35:20.978 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1200 IOs, 491945984 bytes 2024-09-17T23:35:21.085 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1300 IOs, 528305152 bytes 2024-09-17T23:35:21.231 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1400 IOs, 597463040 bytes 2024-09-17T23:35:21.353 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1500 IOs, 640615424 bytes 2024-09-17T23:35:21.468 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1600 IOs, 678848512 bytes 2024-09-17T23:35:21.595 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1700 IOs, 732000256 bytes 2024-09-17T23:35:21.770 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1800 IOs, 840488960 bytes 2024-09-17T23:35:21.887 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1900 IOs, 879527936 bytes 2024-09-17T23:35:21.989 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2000 IOs, 899683328 bytes 2024-09-17T23:35:22.114 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2100 IOs, 952538112 bytes 2024-09-17T23:35:22.286 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2200 IOs, 1066175488 bytes 2024-09-17T23:35:22.398 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2300 IOs, 1080370176 bytes 2024-09-17T23:35:22.533 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2400 IOs, 1138562048 bytes 2024-09-17T23:35:22.680 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2500 IOs, 1212487680 bytes 2024-09-17T23:35:22.789 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2600 IOs, 1240183808 bytes 2024-09-17T23:35:22.871 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-09-17T23:35:22.872 INFO:teuthology.orchestra.run:waiting for 600 2024-09-17T23:35:22.872 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-09-17T23:35:22.872 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-09-17T23:35:22.872 DEBUG:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-09-17T23:35:22.875 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-09-17T23:35:22.875 INFO:tasks.rbd_mirror_thrash:waiting for 106 secs before thrashing 2024-09-17T23:35:22.890 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2700 IOs, 1262610432 bytes 2024-09-17T23:35:23.046 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2800 IOs, 1339204608 bytes 2024-09-17T23:35:23.178 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2900 IOs, 1375423488 bytes 2024-09-17T23:35:23.351 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3000 IOs, 1471567872 bytes 2024-09-17T23:35:23.475 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3100 IOs, 1504956416 bytes 2024-09-17T23:35:23.574 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3200 IOs, 1526037504 bytes 2024-09-17T23:35:23.675 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3300 IOs, 1548065792 bytes 2024-09-17T23:35:23.767 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3400 IOs, 1560221696 bytes 2024-09-17T23:35:23.886 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3500 IOs, 1607553024 bytes 2024-09-17T23:35:24.037 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3600 IOs, 1683387392 bytes 2024-09-17T23:35:24.130 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3700 IOs, 1694657536 bytes 2024-09-17T23:35:24.221 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3800 IOs, 1706811392 bytes 2024-09-17T23:35:24.342 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3900 IOs, 1752721408 bytes 2024-09-17T23:35:24.467 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4000 IOs, 1797081088 bytes 2024-09-17T23:35:24.560 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4100 IOs, 1808618496 bytes 2024-09-17T23:35:24.672 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4200 IOs, 1840393216 bytes 2024-09-17T23:35:24.817 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4300 IOs, 1916121088 bytes 2024-09-17T23:35:24.969 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4400 IOs, 2013773824 bytes 2024-09-17T23:35:25.102 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4500 IOs, 2053700608 bytes 2024-09-17T23:35:25.189 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4600 IOs, 2061387776 bytes 2024-09-17T23:35:25.294 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4700 IOs, 2085548032 bytes 2024-09-17T23:35:25.413 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4800 IOs, 2125011968 bytes 2024-09-17T23:35:25.500 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4900 IOs, 2128053248 bytes 2024-09-17T23:35:25.617 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5000 IOs, 2167207936 bytes 2024-09-17T23:35:25.724 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5100 IOs, 2197555200 bytes 2024-09-17T23:35:25.905 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5200 IOs, 2299966464 bytes 2024-09-17T23:35:26.055 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5300 IOs, 2372615168 bytes 2024-09-17T23:35:26.159 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5400 IOs, 2395738112 bytes 2024-09-17T23:35:26.242 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5500 IOs, 2396379136 bytes 2024-09-17T23:35:26.355 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5600 IOs, 2431622144 bytes 2024-09-17T23:35:26.497 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5700 IOs, 2493332480 bytes 2024-09-17T23:35:26.665 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5800 IOs, 2589706240 bytes 2024-09-17T23:35:26.795 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5900 IOs, 2643759104 bytes 2024-09-17T23:35:26.908 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6000 IOs, 2690667520 bytes 2024-09-17T23:35:27.018 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6100 IOs, 2715509760 bytes 2024-09-17T23:35:27.119 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6200 IOs, 2727974912 bytes 2024-09-17T23:35:27.238 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6300 IOs, 2748359680 bytes 2024-09-17T23:35:27.359 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6400 IOs, 2790324224 bytes 2024-09-17T23:35:27.530 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6500 IOs, 2887814144 bytes 2024-09-17T23:35:27.631 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6600 IOs, 2910644224 bytes 2024-09-17T23:35:27.757 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6700 IOs, 2958048256 bytes 2024-09-17T23:35:27.878 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6800 IOs, 3003649024 bytes 2024-09-17T23:35:28.024 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6900 IOs, 3075448832 bytes 2024-09-17T23:35:28.122 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7000 IOs, 3087907840 bytes 2024-09-17T23:35:28.251 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7100 IOs, 3133545472 bytes 2024-09-17T23:35:28.396 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7200 IOs, 3206272000 bytes 2024-09-17T23:35:28.547 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7300 IOs, 3277685760 bytes 2024-09-17T23:35:28.656 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7400 IOs, 3302838272 bytes 2024-09-17T23:35:28.747 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7500 IOs, 3314573312 bytes 2024-09-17T23:35:28.861 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7600 IOs, 3346336768 bytes 2024-09-17T23:35:29.002 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7700 IOs, 3431725056 bytes 2024-09-17T23:35:29.104 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7800 IOs, 3432349696 bytes 2024-09-17T23:35:29.243 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7900 IOs, 3515443200 bytes 2024-09-17T23:35:29.386 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8000 IOs, 3557679104 bytes 2024-09-17T23:35:29.528 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8100 IOs, 3623361536 bytes 2024-09-17T23:35:29.624 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8200 IOs, 3638291456 bytes 2024-09-17T23:35:29.756 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8300 IOs, 3691358208 bytes 2024-09-17T23:35:29.868 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8400 IOs, 3725365248 bytes 2024-09-17T23:35:29.951 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8500 IOs, 3732123648 bytes 2024-09-17T23:35:30.094 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8600 IOs, 3796060160 bytes 2024-09-17T23:35:30.201 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8700 IOs, 3823521792 bytes 2024-09-17T23:35:30.335 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8800 IOs, 3884951552 bytes 2024-09-17T23:35:30.416 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8900 IOs, 3885571072 bytes 2024-09-17T23:35:30.556 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9000 IOs, 3956359168 bytes 2024-09-17T23:35:30.659 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9100 IOs, 3971042304 bytes 2024-09-17T23:35:30.765 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9200 IOs, 3996246016 bytes 2024-09-17T23:35:30.887 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9300 IOs, 4036586496 bytes 2024-09-17T23:35:31.028 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9400 IOs, 4112749568 bytes 2024-09-17T23:35:31.183 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9500 IOs, 4182236160 bytes 2024-09-17T23:35:31.308 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9600 IOs, 4240254976 bytes 2024-09-17T23:35:31.387 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ error_code=124 2024-09-17T23:35:31.387 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set -e 2024-09-17T23:35:31.387 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' 124 -eq 124 ']' 2024-09-17T23:35:31.387 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:35:31.387 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_present cluster1 mirror image_3 present 2024-09-17T23:35:31.388 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:35:31.388 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:35:31.388 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_3 2024-09-17T23:35:31.388 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=present 2024-09-17T23:35:31.388 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:35:31.388 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:35:31.388 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:35:31.388 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_3 2024-09-17T23:35:31.389 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:35:31.389 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:35:31.389 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_3 2024-09-17T23:35:31.389 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-09-17T23:35:31.389 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:35:31.433 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image_id=12b53be2abd8 2024-09-17T23:35:31.434 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-17T23:35:31.434 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.1 2024-09-17T23:35:31.535 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_present cluster1 mirror image_3 present 12b53be2abd8 2024-09-17T23:35:31.535 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:35:31.536 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:35:31.536 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_3 2024-09-17T23:35:31.536 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=present 2024-09-17T23:35:31.536 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id=12b53be2abd8 2024-09-17T23:35:31.536 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=deleted 2024-09-17T23:35:31.536 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_image_id 2024-09-17T23:35:31.536 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_3 2024-09-17T23:35:31.536 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:35:31.536 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:35:31.536 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_3 2024-09-17T23:35:31.537 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-09-17T23:35:31.537 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:35:31.580 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_image_id=12b53be2abd8 2024-09-17T23:35:31.580 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n 12b53be2abd8 2024-09-17T23:35:31.581 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -z 12b53be2abd8 -o 12b53be2abd8 = 12b53be2abd8 2024-09-17T23:35:31.581 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_state=present 2024-09-17T23:35:31.581 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test present = present 2024-09-17T23:35:31.581 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:35:31.581 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ disable_mirror cluster2 mirror image_3 2024-09-17T23:35:31.581 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:35:31.581 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:35:31.581 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_3 2024-09-17T23:35:31.581 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_3 2024-09-17T23:35:52.984 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.6789466810711354) > weight (0.25) 2024-09-17T23:35:52.985 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.2612313489717851) > weight (0.25) 2024-09-17T23:35:52.985 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.41084327148449906) > weight (0.25) 2024-09-17T23:35:52.985 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.3 2024-09-17T23:35:52.985 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Sent signal 15 2024-09-17T23:35:52.986 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2024-09-17T23:35:52.986 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:35:52.984+0000 7ff50943a640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 38720) UID: 1000 2024-09-17T23:35:57.045 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi038.stderr:2024-09-17T23:35:57.041+0000 7fb6092b2640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55d74596ab60 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-17T23:35:57.045 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi038.stderr:2024-09-17T23:35:57.041+0000 7fb60fabf640 -1 rbd::mirror::ImageReplayer: 0x55d7450c2000 [3/64771169-8de5-40e2-99f6-6a2c02c69326] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-17T23:35:58.986 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.3 2024-09-17T23:35:58.986 INFO:teuthology.orchestra.run:waiting for 600 2024-09-17T23:35:58.987 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.3 2024-09-17T23:35:58.987 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2024-09-17T23:35:58.987 DEBUG:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2024-09-17T23:35:58.990 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2024-09-17T23:35:58.990 INFO:tasks.rbd_mirror_thrash:waiting for 93 secs before thrashing 2024-09-17T23:36:02.311 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi038.stderr:2024-09-17T23:36:02.309+0000 7efed9bce640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5562631c9800 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=187 2024-09-17T23:36:02.311 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi038.stderr:2024-09-17T23:36:02.309+0000 7efed9bce640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5562631c9800 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=187, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=0ba5654d-47fb-4672-b20c-7359a1340314, primary_snap_id=bb, last_copied_object_number=128, snap_seqs={187=18446744073709551614}] 2024-09-17T23:36:02.321 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Mirroring disabled 2024-09-17T23:36:02.342 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_present cluster1 mirror image_3 deleted 2024-09-17T23:36:02.342 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:36:02.343 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:36:02.343 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_3 2024-09-17T23:36:02.343 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=deleted 2024-09-17T23:36:02.343 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:36:02.343 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:36:02.343 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:36:02.343 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_3 2024-09-17T23:36:02.343 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:36:02.343 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:36:02.343 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_3 2024-09-17T23:36:02.344 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-09-17T23:36:02.344 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:36:02.381 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_3: (2) No such file or directory 2024-09-17T23:36:02.384 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image_id= 2024-09-17T23:36:02.384 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-17T23:36:02.384 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.1 2024-09-17T23:36:02.486 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_present cluster1 mirror image_3 deleted '' 2024-09-17T23:36:02.486 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:36:02.486 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:36:02.486 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_3 2024-09-17T23:36:02.486 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=deleted 2024-09-17T23:36:02.487 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:36:02.487 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=deleted 2024-09-17T23:36:02.487 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_image_id 2024-09-17T23:36:02.487 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_3 2024-09-17T23:36:02.487 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:36:02.487 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:36:02.487 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_3 2024-09-17T23:36:02.487 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-09-17T23:36:02.487 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:36:02.522 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_3: (2) No such file or directory 2024-09-17T23:36:02.526 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_image_id= 2024-09-17T23:36:02.526 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:36:02.526 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test deleted = deleted 2024-09-17T23:36:02.526 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:36:02.526 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ purge_snapshots cluster2 mirror image_3 2024-09-17T23:36:02.526 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:36:02.526 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:36:02.527 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_3 2024-09-17T23:36:02.527 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap purge mirror/image_3 2024-09-17T23:36:02.572 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image_retry cluster2 mirror image_3 2024-09-17T23:36:02.572 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:36:02.572 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:36:02.573 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_3 2024-09-17T23:36:02.573 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-17T23:36:02.573 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0 2024-09-17T23:36:02.573 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image cluster2 mirror image_3 2024-09-17T23:36:02.573 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:36:02.573 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:36:02.574 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_3 2024-09-17T23:36:02.574 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_3 2024-09-17T23:36:02.618 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 rm mirror/image_3 2024-09-17T23:36:02.771 INFO:tasks.ceph.cluster2.osd.0.smithi038.stderr:2024-09-17T23:36:02.769+0000 7fa26a72e640 -1 reset not still connected to 0x55b32fcf1d40 2024-09-17T23:36:02.877 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-17T23:36:02.881 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:36:02.881 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 10` 2024-09-17T23:36:02.881 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_4 2024-09-17T23:36:02.881 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_image cluster2 mirror image_4 512M 2024-09-17T23:36:02.881 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:36:02.881 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:36:02.881 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:36:02.882 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:36:02.882 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_4 2024-09-17T23:36:02.882 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:36:02.882 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local size=128 2024-09-17T23:36:02.882 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n 512M ']' 2024-09-17T23:36:02.882 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ size=512M 2024-09-17T23:36:02.882 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:36:02.882 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_4 2024-09-17T23:36:02.932 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ enable_mirror cluster2 mirror image_4 2024-09-17T23:36:02.932 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:36:02.932 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:36:02.932 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_4 2024-09-17T23:36:02.932 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=snapshot 2024-09-17T23:36:02.932 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_4 snapshot 2024-09-17T23:36:03.333 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Mirroring enabled 2024-09-17T23:36:03.344 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 info mirror/image_4 2024-09-17T23:36:03.381 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'image_4': 2024-09-17T23:36:03.381 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 512 MiB in 128 objects 2024-09-17T23:36:03.382 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:36:03.382 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 1 2024-09-17T23:36:03.382 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 19b5e633d72 2024-09-17T23:36:03.382 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.19b5e633d72 2024-09-17T23:36:03.382 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:36:03.382 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-17T23:36:03.382 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:36:03.382 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:36:03.382 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:36:02 2024 2024-09-17T23:36:03.382 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:36:02 2024 2024-09-17T23:36:03.382 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:36:02 2024 2024-09-17T23:36:03.382 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:36:03.382 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:36:03.382 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: dfcfb3d7-e818-4dd4-8ce9-fadc1c217960 2024-09-17T23:36:03.383 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: true 2024-09-17T23:36:03.384 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ stress_write_image cluster2 mirror image_4 2024-09-17T23:36:03.385 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:36:03.385 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:36:03.385 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_4 2024-09-17T23:36:03.385 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-17T23:36:03.386 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local duration=14 2024-09-17T23:36:03.386 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set +e 2024-09-17T23:36:03.386 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_4 --debug-rbd=20 --debug-journaler=20 2024-09-17T23:36:03.689 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:100 IOs, 95144960 bytes 2024-09-17T23:36:03.845 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:200 IOs, 167420928 bytes 2024-09-17T23:36:03.989 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:300 IOs, 222768128 bytes 2024-09-17T23:36:04.083 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:400 IOs, 231683072 bytes 2024-09-17T23:36:04.199 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:500 IOs, 268888064 bytes 2024-09-17T23:36:04.332 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:600 IOs, 327881728 bytes 2024-09-17T23:36:04.481 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:700 IOs, 394289152 bytes 2024-09-17T23:36:04.570 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:800 IOs, 407042048 bytes 2024-09-17T23:36:04.654 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:900 IOs, 407647232 bytes 2024-09-17T23:36:04.773 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1000 IOs, 463823872 bytes 2024-09-17T23:36:04.903 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1100 IOs, 527457280 bytes 2024-09-17T23:36:05.011 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1200 IOs, 550415360 bytes 2024-09-17T23:36:05.127 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1300 IOs, 607015936 bytes 2024-09-17T23:36:05.257 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1400 IOs, 656596992 bytes 2024-09-17T23:36:05.407 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1500 IOs, 727757824 bytes 2024-09-17T23:36:05.516 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1600 IOs, 754478080 bytes 2024-09-17T23:36:05.642 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1700 IOs, 828325888 bytes 2024-09-17T23:36:05.794 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1800 IOs, 904364032 bytes 2024-09-17T23:36:05.909 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1900 IOs, 914482176 bytes 2024-09-17T23:36:06.028 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2000 IOs, 957145088 bytes 2024-09-17T23:36:06.131 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2100 IOs, 982163456 bytes 2024-09-17T23:36:06.229 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2200 IOs, 1001746432 bytes 2024-09-17T23:36:06.311 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2300 IOs, 1002374144 bytes 2024-09-17T23:36:06.416 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2400 IOs, 1030288384 bytes 2024-09-17T23:36:06.504 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2500 IOs, 1040290816 bytes 2024-09-17T23:36:06.655 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2600 IOs, 1120024576 bytes 2024-09-17T23:36:06.749 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2700 IOs, 1136082944 bytes 2024-09-17T23:36:06.866 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2800 IOs, 1183448064 bytes 2024-09-17T23:36:06.995 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2900 IOs, 1222695936 bytes 2024-09-17T23:36:07.098 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3000 IOs, 1249838080 bytes 2024-09-17T23:36:07.220 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3100 IOs, 1281761280 bytes 2024-09-17T23:36:07.344 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3200 IOs, 1328397312 bytes 2024-09-17T23:36:07.477 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3300 IOs, 1385298944 bytes 2024-09-17T23:36:07.591 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3400 IOs, 1420939264 bytes 2024-09-17T23:36:07.704 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3500 IOs, 1457895424 bytes 2024-09-17T23:36:07.826 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3600 IOs, 1506478080 bytes 2024-09-17T23:36:07.916 INFO:tasks.ceph.cluster2.osd.2.smithi038.stderr:2024-09-17T23:36:07.914+0000 7f1c4b04a640 -1 reset not still connected to 0x561ce224d790 2024-09-17T23:36:07.921 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3700 IOs, 1517223936 bytes 2024-09-17T23:36:08.036 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3800 IOs, 1573272576 bytes 2024-09-17T23:36:08.157 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3900 IOs, 1593262080 bytes 2024-09-17T23:36:08.267 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4000 IOs, 1623549952 bytes 2024-09-17T23:36:08.363 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4100 IOs, 1637002240 bytes 2024-09-17T23:36:08.485 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4200 IOs, 1691458560 bytes 2024-09-17T23:36:08.597 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4300 IOs, 1713179648 bytes 2024-09-17T23:36:08.709 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4400 IOs, 1756694528 bytes 2024-09-17T23:36:08.818 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4500 IOs, 1774447616 bytes 2024-09-17T23:36:08.936 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4600 IOs, 1813001216 bytes 2024-09-17T23:36:09.043 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4700 IOs, 1848722432 bytes 2024-09-17T23:36:09.172 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4800 IOs, 1897170944 bytes 2024-09-17T23:36:09.326 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4900 IOs, 1979781120 bytes 2024-09-17T23:36:09.491 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5000 IOs, 2073292800 bytes 2024-09-17T23:36:09.598 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5100 IOs, 2103589888 bytes 2024-09-17T23:36:09.685 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5200 IOs, 2110665728 bytes 2024-09-17T23:36:09.785 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5300 IOs, 2127340544 bytes 2024-09-17T23:36:09.921 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5400 IOs, 2199849984 bytes 2024-09-17T23:36:10.054 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5500 IOs, 2240753664 bytes 2024-09-17T23:36:10.156 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5600 IOs, 2261673984 bytes 2024-09-17T23:36:10.257 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5700 IOs, 2280589312 bytes 2024-09-17T23:36:10.376 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5800 IOs, 2324153344 bytes 2024-09-17T23:36:10.477 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5900 IOs, 2345843712 bytes 2024-09-17T23:36:10.558 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6000 IOs, 2346449920 bytes 2024-09-17T23:36:10.639 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6100 IOs, 2347085824 bytes 2024-09-17T23:36:10.775 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6200 IOs, 2401840128 bytes 2024-09-17T23:36:10.881 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6300 IOs, 2426692608 bytes 2024-09-17T23:36:11.038 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6400 IOs, 2512274432 bytes 2024-09-17T23:36:11.121 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6500 IOs, 2512863232 bytes 2024-09-17T23:36:11.240 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6600 IOs, 2556123136 bytes 2024-09-17T23:36:11.349 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6700 IOs, 2587650048 bytes 2024-09-17T23:36:11.458 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6800 IOs, 2615002112 bytes 2024-09-17T23:36:11.577 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6900 IOs, 2655150080 bytes 2024-09-17T23:36:11.679 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7000 IOs, 2678478848 bytes 2024-09-17T23:36:11.816 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7100 IOs, 2737637376 bytes 2024-09-17T23:36:11.907 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7200 IOs, 2745230336 bytes 2024-09-17T23:36:12.048 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7300 IOs, 2813367296 bytes 2024-09-17T23:36:12.242 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7400 IOs, 2884442112 bytes 2024-09-17T23:36:12.332 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7500 IOs, 2892575744 bytes 2024-09-17T23:36:12.443 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7600 IOs, 2923745280 bytes 2024-09-17T23:36:12.541 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7700 IOs, 2939969536 bytes 2024-09-17T23:36:12.644 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7800 IOs, 2967505920 bytes 2024-09-17T23:36:12.817 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7900 IOs, 3065466880 bytes 2024-09-17T23:36:13.019 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8000 IOs, 3195780096 bytes 2024-09-17T23:36:13.127 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8100 IOs, 3227672576 bytes 2024-09-17T23:36:13.238 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8200 IOs, 3263152128 bytes 2024-09-17T23:36:13.354 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8300 IOs, 3288312832 bytes 2024-09-17T23:36:13.471 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8400 IOs, 3334493184 bytes 2024-09-17T23:36:13.575 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8500 IOs, 3349699584 bytes 2024-09-17T23:36:13.674 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8600 IOs, 3366551552 bytes 2024-09-17T23:36:13.779 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8700 IOs, 3391341568 bytes 2024-09-17T23:36:13.873 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8800 IOs, 3403554816 bytes 2024-09-17T23:36:14.069 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8900 IOs, 3548849152 bytes 2024-09-17T23:36:14.204 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9000 IOs, 3582862336 bytes 2024-09-17T23:36:14.350 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9100 IOs, 3659752448 bytes 2024-09-17T23:36:14.448 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9200 IOs, 3675650048 bytes 2024-09-17T23:36:14.532 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9300 IOs, 3680609280 bytes 2024-09-17T23:36:14.637 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9400 IOs, 3704137728 bytes 2024-09-17T23:36:14.769 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9500 IOs, 3761468416 bytes 2024-09-17T23:36:14.867 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9600 IOs, 3779673088 bytes 2024-09-17T23:36:15.011 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9700 IOs, 3842626560 bytes 2024-09-17T23:36:15.110 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9800 IOs, 3866119168 bytes 2024-09-17T23:36:15.240 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9900 IOs, 3917200384 bytes 2024-09-17T23:36:15.376 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10000 IOs, 3990564864 bytes 2024-09-17T23:36:15.508 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10100 IOs, 4031072256 bytes 2024-09-17T23:36:15.646 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10200 IOs, 4095077376 bytes 2024-09-17T23:36:15.772 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10300 IOs, 4145344512 bytes 2024-09-17T23:36:15.872 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10400 IOs, 4163651584 bytes 2024-09-17T23:36:16.009 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10500 IOs, 4224696320 bytes 2024-09-17T23:36:16.276 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10600 IOs, 4225323008 bytes 2024-09-17T23:36:16.418 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10700 IOs, 4294679552 bytes 2024-09-17T23:36:16.512 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10800 IOs, 4306731008 bytes 2024-09-17T23:36:16.618 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10900 IOs, 4332592128 bytes 2024-09-17T23:36:16.705 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:11000 IOs, 4340238336 bytes 2024-09-17T23:36:16.803 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:11100 IOs, 4359232512 bytes 2024-09-17T23:36:16.906 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:11200 IOs, 4381655040 bytes 2024-09-17T23:36:17.000 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:11300 IOs, 4395768832 bytes 2024-09-17T23:36:17.113 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:11400 IOs, 4426101760 bytes 2024-09-17T23:36:17.243 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:11500 IOs, 4488071168 bytes 2024-09-17T23:36:17.393 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ error_code=124 2024-09-17T23:36:17.393 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set -e 2024-09-17T23:36:17.393 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' 124 -eq 124 ']' 2024-09-17T23:36:17.393 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:36:17.393 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_present cluster1 mirror image_4 present 2024-09-17T23:36:17.393 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:36:17.393 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:36:17.394 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_4 2024-09-17T23:36:17.394 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=present 2024-09-17T23:36:17.394 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:36:17.394 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:36:17.394 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:36:17.394 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_4 2024-09-17T23:36:17.394 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:36:17.394 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:36:17.394 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_4 2024-09-17T23:36:17.394 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-09-17T23:36:17.394 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:36:17.438 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image_id=12b59ad90d66 2024-09-17T23:36:17.438 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-17T23:36:17.438 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.1 2024-09-17T23:36:17.539 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_present cluster1 mirror image_4 present 12b59ad90d66 2024-09-17T23:36:17.540 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:36:17.540 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:36:17.540 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_4 2024-09-17T23:36:17.540 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=present 2024-09-17T23:36:17.540 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id=12b59ad90d66 2024-09-17T23:36:17.540 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=deleted 2024-09-17T23:36:17.540 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_image_id 2024-09-17T23:36:17.540 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_4 2024-09-17T23:36:17.540 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:36:17.540 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:36:17.540 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_4 2024-09-17T23:36:17.540 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-09-17T23:36:17.541 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:36:17.585 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_image_id=12b59ad90d66 2024-09-17T23:36:17.585 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n 12b59ad90d66 2024-09-17T23:36:17.585 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -z 12b59ad90d66 -o 12b59ad90d66 = 12b59ad90d66 2024-09-17T23:36:17.585 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_state=present 2024-09-17T23:36:17.585 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test present = present 2024-09-17T23:36:17.585 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:36:17.585 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ disable_mirror cluster2 mirror image_4 2024-09-17T23:36:17.585 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:36:17.585 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:36:17.586 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_4 2024-09-17T23:36:17.586 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_4 2024-09-17T23:36:25.323 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi038.stderr:2024-09-17T23:36:25.321+0000 7fb609ab3640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55d74596b040 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-17T23:36:25.323 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi038.stderr:2024-09-17T23:36:25.321+0000 7fb60fabf640 -1 rbd::mirror::ImageReplayer: 0x55d745a44a00 [3/dfcfb3d7-e818-4dd4-8ce9-fadc1c217960] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-17T23:36:47.327 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Mirroring disabled 2024-09-17T23:36:47.349 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_present cluster1 mirror image_4 deleted 2024-09-17T23:36:47.350 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:36:47.350 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:36:47.350 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_4 2024-09-17T23:36:47.350 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=deleted 2024-09-17T23:36:47.350 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:36:47.350 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:36:47.350 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:36:47.350 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_4 2024-09-17T23:36:47.350 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:36:47.350 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:36:47.350 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_4 2024-09-17T23:36:47.350 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-09-17T23:36:47.350 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:36:47.388 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_4: (2) No such file or directory 2024-09-17T23:36:47.391 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image_id= 2024-09-17T23:36:47.391 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-17T23:36:47.391 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.1 2024-09-17T23:36:47.493 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_present cluster1 mirror image_4 deleted '' 2024-09-17T23:36:47.493 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:36:47.493 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:36:47.493 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_4 2024-09-17T23:36:47.493 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=deleted 2024-09-17T23:36:47.493 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:36:47.493 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=deleted 2024-09-17T23:36:47.493 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_image_id 2024-09-17T23:36:47.493 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_4 2024-09-17T23:36:47.494 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:36:47.494 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:36:47.494 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_4 2024-09-17T23:36:47.494 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-09-17T23:36:47.494 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:36:47.529 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_4: (2) No such file or directory 2024-09-17T23:36:47.533 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_image_id= 2024-09-17T23:36:47.533 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:36:47.533 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test deleted = deleted 2024-09-17T23:36:47.533 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:36:47.533 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ purge_snapshots cluster2 mirror image_4 2024-09-17T23:36:47.533 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:36:47.533 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:36:47.533 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_4 2024-09-17T23:36:47.534 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap purge mirror/image_4 2024-09-17T23:36:47.579 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image_retry cluster2 mirror image_4 2024-09-17T23:36:47.579 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:36:47.579 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:36:47.579 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_4 2024-09-17T23:36:47.580 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-17T23:36:47.580 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0 2024-09-17T23:36:47.580 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image cluster2 mirror image_4 2024-09-17T23:36:47.580 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:36:47.580 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:36:47.581 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_4 2024-09-17T23:36:47.581 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_4 2024-09-17T23:36:47.625 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 rm mirror/image_4 2024-09-17T23:36:47.892 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-17T23:36:47.896 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:36:47.896 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 10` 2024-09-17T23:36:47.896 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_5 2024-09-17T23:36:47.896 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_image cluster2 mirror image_5 512M 2024-09-17T23:36:47.896 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:36:47.896 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:36:47.897 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:36:47.897 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:36:47.897 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_5 2024-09-17T23:36:47.897 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:36:47.897 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local size=128 2024-09-17T23:36:47.897 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n 512M ']' 2024-09-17T23:36:47.897 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ size=512M 2024-09-17T23:36:47.897 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:36:47.897 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_5 2024-09-17T23:36:47.947 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ enable_mirror cluster2 mirror image_5 2024-09-17T23:36:47.947 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:36:47.947 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:36:47.947 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_5 2024-09-17T23:36:47.947 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=snapshot 2024-09-17T23:36:47.948 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_5 snapshot 2024-09-17T23:36:48.319 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Mirroring enabled 2024-09-17T23:36:48.329 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 info mirror/image_5 2024-09-17T23:36:48.367 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'image_5': 2024-09-17T23:36:48.367 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 512 MiB in 128 objects 2024-09-17T23:36:48.367 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:36:48.368 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 1 2024-09-17T23:36:48.368 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 19cd6cd451db 2024-09-17T23:36:48.368 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.19cd6cd451db 2024-09-17T23:36:48.368 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:36:48.368 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-17T23:36:48.368 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:36:48.368 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:36:48.368 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:36:47 2024 2024-09-17T23:36:48.368 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:36:47 2024 2024-09-17T23:36:48.368 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:36:47 2024 2024-09-17T23:36:48.368 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:36:48.368 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:36:48.368 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: 3ac77a55-dd26-41fd-b7dd-85fd8a526205 2024-09-17T23:36:48.369 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: true 2024-09-17T23:36:48.371 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ stress_write_image cluster2 mirror image_5 2024-09-17T23:36:48.371 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:36:48.371 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:36:48.371 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_5 2024-09-17T23:36:48.371 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-17T23:36:48.373 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local duration=12 2024-09-17T23:36:48.373 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set +e 2024-09-17T23:36:48.373 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_5 --debug-rbd=20 --debug-journaler=20 2024-09-17T23:36:48.828 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:100 IOs, 45345792 bytes 2024-09-17T23:36:48.954 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:200 IOs, 53705728 bytes 2024-09-17T23:36:49.084 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:300 IOs, 91057152 bytes 2024-09-17T23:36:49.211 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:400 IOs, 133935104 bytes 2024-09-17T23:36:49.381 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:500 IOs, 199359488 bytes 2024-09-17T23:36:49.510 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:600 IOs, 248025088 bytes 2024-09-17T23:36:49.699 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:700 IOs, 318732288 bytes 2024-09-17T23:36:49.781 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:800 IOs, 319364096 bytes 2024-09-17T23:36:49.928 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:900 IOs, 384057344 bytes 2024-09-17T23:36:50.042 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1000 IOs, 419179520 bytes 2024-09-17T23:36:50.181 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1100 IOs, 467869696 bytes 2024-09-17T23:36:50.348 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1200 IOs, 554854400 bytes 2024-09-17T23:36:50.469 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1300 IOs, 594242560 bytes 2024-09-17T23:36:50.581 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1400 IOs, 624059392 bytes 2024-09-17T23:36:50.682 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1500 IOs, 643247104 bytes 2024-09-17T23:36:50.770 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1600 IOs, 666933248 bytes 2024-09-17T23:36:50.956 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1700 IOs, 755496960 bytes 2024-09-17T23:36:51.063 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1800 IOs, 795310080 bytes 2024-09-17T23:36:51.162 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1900 IOs, 803632128 bytes 2024-09-17T23:36:51.316 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2000 IOs, 880984064 bytes 2024-09-17T23:36:51.415 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2100 IOs, 903411712 bytes 2024-09-17T23:36:51.561 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2200 IOs, 977090560 bytes 2024-09-17T23:36:51.695 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2300 IOs, 1041001472 bytes 2024-09-17T23:36:51.782 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2400 IOs, 1049273344 bytes 2024-09-17T23:36:51.889 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2500 IOs, 1070454784 bytes 2024-09-17T23:36:52.053 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2600 IOs, 1161848832 bytes 2024-09-17T23:36:52.165 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2700 IOs, 1198347264 bytes 2024-09-17T23:36:52.287 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2800 IOs, 1222984704 bytes 2024-09-17T23:36:52.391 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2900 IOs, 1247297536 bytes 2024-09-17T23:36:52.491 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3000 IOs, 1274902528 bytes 2024-09-17T23:36:52.592 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3100 IOs, 1290037248 bytes 2024-09-17T23:36:52.692 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3200 IOs, 1309231104 bytes 2024-09-17T23:36:52.840 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3300 IOs, 1384403968 bytes 2024-09-17T23:36:52.946 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3400 IOs, 1410792448 bytes 2024-09-17T23:36:53.092 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3500 IOs, 1480564736 bytes 2024-09-17T23:36:53.180 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3600 IOs, 1481218048 bytes 2024-09-17T23:36:53.330 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3700 IOs, 1553085440 bytes 2024-09-17T23:36:53.445 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3800 IOs, 1584303104 bytes 2024-09-17T23:36:53.566 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3900 IOs, 1630489600 bytes 2024-09-17T23:36:53.668 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4000 IOs, 1655093248 bytes 2024-09-17T23:36:53.797 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4100 IOs, 1705442304 bytes 2024-09-17T23:36:53.889 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4200 IOs, 1723655168 bytes 2024-09-17T23:36:53.979 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4300 IOs, 1724270592 bytes 2024-09-17T23:36:54.100 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4400 IOs, 1782632448 bytes 2024-09-17T23:36:54.234 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4500 IOs, 1817544704 bytes 2024-09-17T23:36:54.341 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4600 IOs, 1846343680 bytes 2024-09-17T23:36:54.536 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4700 IOs, 1976552448 bytes 2024-09-17T23:36:54.684 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4800 IOs, 2043067392 bytes 2024-09-17T23:36:54.801 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4900 IOs, 2080319488 bytes 2024-09-17T23:36:54.923 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5000 IOs, 2127780864 bytes 2024-09-17T23:36:55.042 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5100 IOs, 2169857024 bytes 2024-09-17T23:36:55.151 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5200 IOs, 2213745664 bytes 2024-09-17T23:36:55.311 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5300 IOs, 2271940608 bytes 2024-09-17T23:36:55.441 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5400 IOs, 2323602432 bytes 2024-09-17T23:36:55.524 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5500 IOs, 2324226048 bytes 2024-09-17T23:36:55.655 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5600 IOs, 2391521280 bytes 2024-09-17T23:36:55.801 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5700 IOs, 2449264640 bytes 2024-09-17T23:36:55.926 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5800 IOs, 2501325824 bytes 2024-09-17T23:36:56.068 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5900 IOs, 2558231552 bytes 2024-09-17T23:36:56.166 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6000 IOs, 2574551040 bytes 2024-09-17T23:36:56.279 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6100 IOs, 2605860864 bytes 2024-09-17T23:36:56.420 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6200 IOs, 2670386176 bytes 2024-09-17T23:36:56.558 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6300 IOs, 2730602496 bytes 2024-09-17T23:36:56.658 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6400 IOs, 2751218688 bytes 2024-09-17T23:36:56.814 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6500 IOs, 2844568576 bytes 2024-09-17T23:36:56.946 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6600 IOs, 2887830528 bytes 2024-09-17T23:36:57.088 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6700 IOs, 2972414976 bytes 2024-09-17T23:36:57.219 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6800 IOs, 3003991040 bytes 2024-09-17T23:36:57.311 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6900 IOs, 3014042624 bytes 2024-09-17T23:36:57.395 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7000 IOs, 3014664192 bytes 2024-09-17T23:36:57.496 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7100 IOs, 3033632768 bytes 2024-09-17T23:36:57.598 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7200 IOs, 3058207744 bytes 2024-09-17T23:36:57.709 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7300 IOs, 3086128128 bytes 2024-09-17T23:36:57.853 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7400 IOs, 3159754752 bytes 2024-09-17T23:36:58.003 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7500 IOs, 3213701120 bytes 2024-09-17T23:36:58.191 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7600 IOs, 3325272064 bytes 2024-09-17T23:36:58.345 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7700 IOs, 3392524288 bytes 2024-09-17T23:36:58.456 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7800 IOs, 3425028096 bytes 2024-09-17T23:36:58.590 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7900 IOs, 3477396480 bytes 2024-09-17T23:36:58.739 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8000 IOs, 3552464896 bytes 2024-09-17T23:36:58.832 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8100 IOs, 3561881600 bytes 2024-09-17T23:36:58.964 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8200 IOs, 3615045632 bytes 2024-09-17T23:36:59.057 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8300 IOs, 3626693632 bytes 2024-09-17T23:36:59.178 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8400 IOs, 3679366144 bytes 2024-09-17T23:36:59.282 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8500 IOs, 3686079488 bytes 2024-09-17T23:36:59.421 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8600 IOs, 3742759936 bytes 2024-09-17T23:36:59.581 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8700 IOs, 3825679360 bytes 2024-09-17T23:36:59.687 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8800 IOs, 3854219264 bytes 2024-09-17T23:36:59.838 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8900 IOs, 3932410880 bytes 2024-09-17T23:36:59.952 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9000 IOs, 3965707264 bytes 2024-09-17T23:37:00.079 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9100 IOs, 4014082048 bytes 2024-09-17T23:37:00.380 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ error_code=124 2024-09-17T23:37:00.381 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set -e 2024-09-17T23:37:00.381 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' 124 -eq 124 ']' 2024-09-17T23:37:00.381 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:37:00.381 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_present cluster1 mirror image_5 present 2024-09-17T23:37:00.381 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:37:00.381 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:37:00.381 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_5 2024-09-17T23:37:00.381 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=present 2024-09-17T23:37:00.381 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:37:00.381 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:37:00.381 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:37:00.382 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_5 2024-09-17T23:37:00.382 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:37:00.382 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:37:00.382 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_5 2024-09-17T23:37:00.383 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-09-17T23:37:00.383 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:37:00.430 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image_id=12b5ebe95934 2024-09-17T23:37:00.430 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-17T23:37:00.430 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.1 2024-09-17T23:37:00.532 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_present cluster1 mirror image_5 present 12b5ebe95934 2024-09-17T23:37:00.532 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:37:00.532 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:37:00.532 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_5 2024-09-17T23:37:00.532 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=present 2024-09-17T23:37:00.532 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id=12b5ebe95934 2024-09-17T23:37:00.532 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=deleted 2024-09-17T23:37:00.532 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_image_id 2024-09-17T23:37:00.532 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_5 2024-09-17T23:37:00.533 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:37:00.533 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:37:00.533 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_5 2024-09-17T23:37:00.533 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-09-17T23:37:00.533 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:37:00.578 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_image_id=12b5ebe95934 2024-09-17T23:37:00.578 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n 12b5ebe95934 2024-09-17T23:37:00.578 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -z 12b5ebe95934 -o 12b5ebe95934 = 12b5ebe95934 2024-09-17T23:37:00.578 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_state=present 2024-09-17T23:37:00.578 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test present = present 2024-09-17T23:37:00.578 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:37:00.578 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ disable_mirror cluster2 mirror image_5 2024-09-17T23:37:00.578 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:37:00.578 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:37:00.578 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_5 2024-09-17T23:37:00.579 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_5 2024-09-17T23:37:08.876 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.6394470025762943) > weight (0.25) 2024-09-17T23:37:08.876 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.7775309647220761) > weight (0.25) 2024-09-17T23:37:08.876 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.9270353566466426) > weight (0.25) 2024-09-17T23:37:08.877 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.8043824425695557) > weight (0.25) 2024-09-17T23:37:08.877 INFO:tasks.rbd_mirror_thrash:waiting for 66 secs before thrashing 2024-09-17T23:37:25.323 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi038.stderr:2024-09-17T23:37:25.321+0000 7fb6092b2640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55d74596b520 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-17T23:37:25.324 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi038.stderr:2024-09-17T23:37:25.321+0000 7fb60fabf640 -1 rbd::mirror::ImageReplayer: 0x55d7450c2000 [3/3ac77a55-dd26-41fd-b7dd-85fd8a526205] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-17T23:37:31.250 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi038.stderr:2024-09-17T23:37:31.248+0000 7efed93cd640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x556264d62400 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=189 2024-09-17T23:37:31.250 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi038.stderr:2024-09-17T23:37:31.248+0000 7efed93cd640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x556264d62400 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=189, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=0ba5654d-47fb-4672-b20c-7359a1340314, primary_snap_id=bd, last_copied_object_number=128, snap_seqs={189=18446744073709551614}] 2024-09-17T23:37:31.258 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Mirroring disabled 2024-09-17T23:37:31.284 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_present cluster1 mirror image_5 deleted 2024-09-17T23:37:31.284 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:37:31.284 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:37:31.284 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_5 2024-09-17T23:37:31.284 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=deleted 2024-09-17T23:37:31.284 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:37:31.284 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:37:31.284 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:37:31.284 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_5 2024-09-17T23:37:31.285 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:37:31.285 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:37:31.285 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_5 2024-09-17T23:37:31.285 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-09-17T23:37:31.285 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:37:31.321 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_5: (2) No such file or directory 2024-09-17T23:37:31.325 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image_id= 2024-09-17T23:37:31.325 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-17T23:37:31.325 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.1 2024-09-17T23:37:31.427 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_present cluster1 mirror image_5 deleted '' 2024-09-17T23:37:31.427 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:37:31.427 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:37:31.427 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_5 2024-09-17T23:37:31.427 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=deleted 2024-09-17T23:37:31.427 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:37:31.427 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=deleted 2024-09-17T23:37:31.428 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_image_id 2024-09-17T23:37:31.428 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_5 2024-09-17T23:37:31.428 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:37:31.428 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:37:31.428 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_5 2024-09-17T23:37:31.428 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-09-17T23:37:31.428 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:37:31.462 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_5: (2) No such file or directory 2024-09-17T23:37:31.466 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_image_id= 2024-09-17T23:37:31.466 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:37:31.466 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test deleted = deleted 2024-09-17T23:37:31.466 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:37:31.466 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ purge_snapshots cluster2 mirror image_5 2024-09-17T23:37:31.466 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:37:31.466 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:37:31.466 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_5 2024-09-17T23:37:31.467 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap purge mirror/image_5 2024-09-17T23:37:31.511 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image_retry cluster2 mirror image_5 2024-09-17T23:37:31.511 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:37:31.511 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:37:31.511 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_5 2024-09-17T23:37:31.511 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-17T23:37:31.511 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0 2024-09-17T23:37:31.512 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image cluster2 mirror image_5 2024-09-17T23:37:31.512 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:37:31.512 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:37:31.512 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_5 2024-09-17T23:37:31.512 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_5 2024-09-17T23:37:31.556 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 rm mirror/image_5 2024-09-17T23:37:31.819 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-17T23:37:31.823 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:37:31.823 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 10` 2024-09-17T23:37:31.823 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_6 2024-09-17T23:37:31.823 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_image cluster2 mirror image_6 512M 2024-09-17T23:37:31.823 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:37:31.823 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:37:31.823 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:37:31.823 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:37:31.823 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_6 2024-09-17T23:37:31.824 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:37:31.824 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local size=128 2024-09-17T23:37:31.824 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n 512M ']' 2024-09-17T23:37:31.824 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ size=512M 2024-09-17T23:37:31.824 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:37:31.824 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_6 2024-09-17T23:37:31.874 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ enable_mirror cluster2 mirror image_6 2024-09-17T23:37:31.874 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:37:31.874 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:37:31.874 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_6 2024-09-17T23:37:31.874 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=snapshot 2024-09-17T23:37:31.874 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_6 snapshot 2024-09-17T23:37:31.990 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.4914023371794428) > weight (0.25) 2024-09-17T23:37:31.990 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.48882296804821723) > weight (0.25) 2024-09-17T23:37:31.991 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.5032180237265218) > weight (0.25) 2024-09-17T23:37:31.991 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.3 2024-09-17T23:37:31.991 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Sent signal 15 2024-09-17T23:37:31.991 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2024-09-17T23:37:31.992 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:37:31.991+0000 7f8f9a100640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 68607) UID: 1000 2024-09-17T23:37:32.260 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Mirroring enabled 2024-09-17T23:37:32.270 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 info mirror/image_6 2024-09-17T23:37:32.307 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'image_6': 2024-09-17T23:37:32.308 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 512 MiB in 128 objects 2024-09-17T23:37:32.308 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:37:32.308 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 1 2024-09-17T23:37:32.308 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 19e556ee1f63 2024-09-17T23:37:32.308 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.19e556ee1f63 2024-09-17T23:37:32.308 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:37:32.308 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-17T23:37:32.308 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:37:32.308 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:37:32.308 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:37:31 2024 2024-09-17T23:37:32.308 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:37:31 2024 2024-09-17T23:37:32.308 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:37:31 2024 2024-09-17T23:37:32.309 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:37:32.309 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:37:32.309 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: 05403773-f07f-439d-9ece-fc2aba7998c9 2024-09-17T23:37:32.309 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: true 2024-09-17T23:37:32.311 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ stress_write_image cluster2 mirror image_6 2024-09-17T23:37:32.311 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:37:32.312 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:37:32.312 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_6 2024-09-17T23:37:32.312 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-17T23:37:32.313 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local duration=13 2024-09-17T23:37:32.314 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set +e 2024-09-17T23:37:32.314 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_6 --debug-rbd=20 --debug-journaler=20 2024-09-17T23:37:32.608 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:100 IOs, 81677312 bytes 2024-09-17T23:37:32.775 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:200 IOs, 123006976 bytes 2024-09-17T23:37:32.867 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:300 IOs, 123778048 bytes 2024-09-17T23:37:32.971 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:400 IOs, 148132864 bytes 2024-09-17T23:37:33.063 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:500 IOs, 162174976 bytes 2024-09-17T23:37:33.211 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:600 IOs, 249752576 bytes 2024-09-17T23:37:33.357 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:700 IOs, 313521152 bytes 2024-09-17T23:37:33.489 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:800 IOs, 366134272 bytes 2024-09-17T23:37:33.614 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:900 IOs, 424351744 bytes 2024-09-17T23:37:33.806 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1000 IOs, 540141568 bytes 2024-09-17T23:37:33.897 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1100 IOs, 569976832 bytes 2024-09-17T23:37:34.035 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1200 IOs, 618463232 bytes 2024-09-17T23:37:34.160 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1300 IOs, 680544256 bytes 2024-09-17T23:37:34.303 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1400 IOs, 741309440 bytes 2024-09-17T23:37:34.497 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1500 IOs, 876103680 bytes 2024-09-17T23:37:34.605 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1600 IOs, 908930048 bytes 2024-09-17T23:37:34.731 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1700 IOs, 945061888 bytes 2024-09-17T23:37:34.862 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1800 IOs, 1006222336 bytes 2024-09-17T23:37:34.992 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1900 IOs, 1059401728 bytes 2024-09-17T23:37:35.091 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2000 IOs, 1080054784 bytes 2024-09-17T23:37:35.175 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2100 IOs, 1080671232 bytes 2024-09-17T23:37:35.306 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2200 IOs, 1132951552 bytes 2024-09-17T23:37:35.398 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2300 IOs, 1151142912 bytes 2024-09-17T23:37:35.484 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2400 IOs, 1151754240 bytes 2024-09-17T23:37:35.610 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2500 IOs, 1199000576 bytes 2024-09-17T23:37:35.701 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2600 IOs, 1209689088 bytes 2024-09-17T23:37:35.813 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2700 IOs, 1247398912 bytes 2024-09-17T23:37:35.899 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2800 IOs, 1252454400 bytes 2024-09-17T23:37:35.992 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.3 2024-09-17T23:37:35.992 INFO:teuthology.orchestra.run:waiting for 600 2024-09-17T23:37:35.993 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.3 2024-09-17T23:37:35.993 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2024-09-17T23:37:35.993 DEBUG:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2024-09-17T23:37:35.996 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2024-09-17T23:37:35.996 INFO:tasks.rbd_mirror_thrash:waiting for 89 secs before thrashing 2024-09-17T23:37:36.006 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2900 IOs, 1279462400 bytes 2024-09-17T23:37:36.151 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3000 IOs, 1331433472 bytes 2024-09-17T23:37:36.275 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3100 IOs, 1372510208 bytes 2024-09-17T23:37:36.377 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3200 IOs, 1373112320 bytes 2024-09-17T23:37:36.490 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3300 IOs, 1401396224 bytes 2024-09-17T23:37:36.614 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3400 IOs, 1449375744 bytes 2024-09-17T23:37:36.699 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3500 IOs, 1450008576 bytes 2024-09-17T23:37:36.848 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3600 IOs, 1522369536 bytes 2024-09-17T23:37:36.959 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3700 IOs, 1553093632 bytes 2024-09-17T23:37:37.053 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3800 IOs, 1565009920 bytes 2024-09-17T23:37:37.147 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3900 IOs, 1579907072 bytes 2024-09-17T23:37:37.298 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4000 IOs, 1663680512 bytes 2024-09-17T23:37:37.410 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4100 IOs, 1679772672 bytes 2024-09-17T23:37:37.576 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4200 IOs, 1775265792 bytes 2024-09-17T23:37:37.702 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4300 IOs, 1828330496 bytes 2024-09-17T23:37:37.849 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4400 IOs, 1902051328 bytes 2024-09-17T23:37:37.964 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4500 IOs, 1934959616 bytes 2024-09-17T23:37:38.075 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4600 IOs, 1968785408 bytes 2024-09-17T23:37:38.175 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4700 IOs, 1985903616 bytes 2024-09-17T23:37:38.288 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4800 IOs, 2031214592 bytes 2024-09-17T23:37:38.413 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4900 IOs, 2060149760 bytes 2024-09-17T23:37:38.513 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5000 IOs, 2083938304 bytes 2024-09-17T23:37:38.619 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5100 IOs, 2108208128 bytes 2024-09-17T23:37:38.761 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5200 IOs, 2178886656 bytes 2024-09-17T23:37:38.907 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5300 IOs, 2249182208 bytes 2024-09-17T23:37:39.008 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5400 IOs, 2272893952 bytes 2024-09-17T23:37:39.115 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5500 IOs, 2302731264 bytes 2024-09-17T23:37:39.226 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5600 IOs, 2331828224 bytes 2024-09-17T23:37:39.374 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5700 IOs, 2402278400 bytes 2024-09-17T23:37:39.486 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5800 IOs, 2442212352 bytes 2024-09-17T23:37:39.608 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5900 IOs, 2487755776 bytes 2024-09-17T23:37:39.714 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6000 IOs, 2511953920 bytes 2024-09-17T23:37:39.809 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6100 IOs, 2528338944 bytes 2024-09-17T23:37:39.892 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6200 IOs, 2528967680 bytes 2024-09-17T23:37:39.988 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6300 IOs, 2547475456 bytes 2024-09-17T23:37:40.119 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6400 IOs, 2598474752 bytes 2024-09-17T23:37:40.214 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6500 IOs, 2613772288 bytes 2024-09-17T23:37:40.326 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6600 IOs, 2643424256 bytes 2024-09-17T23:37:40.447 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6700 IOs, 2686266368 bytes 2024-09-17T23:37:40.598 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6800 IOs, 2779520000 bytes 2024-09-17T23:37:40.783 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6900 IOs, 2833289216 bytes 2024-09-17T23:37:40.916 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7000 IOs, 2883649536 bytes 2024-09-17T23:37:41.058 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7100 IOs, 2949661696 bytes 2024-09-17T23:37:41.163 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7200 IOs, 2971970560 bytes 2024-09-17T23:37:41.329 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7300 IOs, 3064529920 bytes 2024-09-17T23:37:41.449 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7400 IOs, 3107666944 bytes 2024-09-17T23:37:41.550 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7500 IOs, 3128573952 bytes 2024-09-17T23:37:41.589 INFO:tasks.ceph.cluster2.osd.2.smithi038.stderr:2024-09-17T23:37:41.587+0000 7f1c4b04a640 -1 reset not still connected to 0x561cd21728f0 2024-09-17T23:37:41.674 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7600 IOs, 3176571904 bytes 2024-09-17T23:37:41.777 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7700 IOs, 3220740096 bytes 2024-09-17T23:37:41.898 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7800 IOs, 3246186496 bytes 2024-09-17T23:37:42.035 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:7900 IOs, 3313806336 bytes 2024-09-17T23:37:42.162 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8000 IOs, 3365403648 bytes 2024-09-17T23:37:42.276 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8100 IOs, 3397018624 bytes 2024-09-17T23:37:42.374 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8200 IOs, 3420044288 bytes 2024-09-17T23:37:42.502 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8300 IOs, 3483814912 bytes 2024-09-17T23:37:42.664 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8400 IOs, 3556679680 bytes 2024-09-17T23:37:42.802 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8500 IOs, 3620675584 bytes 2024-09-17T23:37:42.934 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8600 IOs, 3673392128 bytes 2024-09-17T23:37:43.020 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8700 IOs, 3679704064 bytes 2024-09-17T23:37:43.126 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8800 IOs, 3704117248 bytes 2024-09-17T23:37:43.226 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:8900 IOs, 3726170112 bytes 2024-09-17T23:37:43.327 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9000 IOs, 3746678784 bytes 2024-09-17T23:37:43.436 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9100 IOs, 3780042752 bytes 2024-09-17T23:37:43.578 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9200 IOs, 3842840576 bytes 2024-09-17T23:37:43.676 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9300 IOs, 3862453248 bytes 2024-09-17T23:37:43.793 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9400 IOs, 3888250880 bytes 2024-09-17T23:37:43.926 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9500 IOs, 3945238528 bytes 2024-09-17T23:37:44.008 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9600 IOs, 3945870336 bytes 2024-09-17T23:37:44.121 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9700 IOs, 3977384960 bytes 2024-09-17T23:37:44.244 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9800 IOs, 4018273280 bytes 2024-09-17T23:37:44.392 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:9900 IOs, 4089634816 bytes 2024-09-17T23:37:44.476 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10000 IOs, 4093815808 bytes 2024-09-17T23:37:44.564 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10100 IOs, 4103826432 bytes 2024-09-17T23:37:44.685 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10200 IOs, 4141599744 bytes 2024-09-17T23:37:44.802 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10300 IOs, 4179852288 bytes 2024-09-17T23:37:45.089 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10400 IOs, 4198237184 bytes 2024-09-17T23:37:45.205 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:10500 IOs, 4233372672 bytes 2024-09-17T23:37:45.321 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ error_code=124 2024-09-17T23:37:45.322 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set -e 2024-09-17T23:37:45.322 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' 124 -eq 124 ']' 2024-09-17T23:37:45.322 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:37:45.322 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_present cluster1 mirror image_6 present 2024-09-17T23:37:45.322 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:37:45.322 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:37:45.322 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_6 2024-09-17T23:37:45.322 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=present 2024-09-17T23:37:45.322 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:37:45.322 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:37:45.322 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:37:45.322 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_6 2024-09-17T23:37:45.322 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:37:45.322 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:37:45.323 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_6 2024-09-17T23:37:45.323 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-09-17T23:37:45.323 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:37:45.366 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image_id=12b5395efbb2 2024-09-17T23:37:45.366 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-17T23:37:45.366 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.1 2024-09-17T23:37:45.468 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_present cluster1 mirror image_6 present 12b5395efbb2 2024-09-17T23:37:45.468 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:37:45.468 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:37:45.468 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_6 2024-09-17T23:37:45.468 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=present 2024-09-17T23:37:45.468 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id=12b5395efbb2 2024-09-17T23:37:45.468 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=deleted 2024-09-17T23:37:45.468 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_image_id 2024-09-17T23:37:45.468 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_6 2024-09-17T23:37:45.468 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:37:45.468 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:37:45.469 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_6 2024-09-17T23:37:45.469 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-09-17T23:37:45.469 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:37:45.515 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_image_id=12b5395efbb2 2024-09-17T23:37:45.515 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n 12b5395efbb2 2024-09-17T23:37:45.515 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -z 12b5395efbb2 -o 12b5395efbb2 = 12b5395efbb2 2024-09-17T23:37:45.515 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_state=present 2024-09-17T23:37:45.515 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test present = present 2024-09-17T23:37:45.516 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:37:45.516 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ disable_mirror cluster2 mirror image_6 2024-09-17T23:37:45.516 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:37:45.516 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:37:45.516 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_6 2024-09-17T23:37:45.516 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_6 2024-09-17T23:37:55.323 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi038.stderr:2024-09-17T23:37:55.321+0000 7fb609ab3640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55d744fc0000 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-17T23:37:55.323 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi038.stderr:2024-09-17T23:37:55.321+0000 7fb6102c0640 -1 rbd::mirror::ImageReplayer: 0x55d745a44a00 [3/05403773-f07f-439d-9ece-fc2aba7998c9] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-17T23:38:14.326 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi038.stderr:2024-09-17T23:38:14.323+0000 7efed93cd640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55626532a000 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=190 2024-09-17T23:38:14.326 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi038.stderr:2024-09-17T23:38:14.323+0000 7efed93cd640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55626532a000 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=190, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=0ba5654d-47fb-4672-b20c-7359a1340314, primary_snap_id=be, last_copied_object_number=128, snap_seqs={190=18446744073709551614}] 2024-09-17T23:38:14.333 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Mirroring disabled 2024-09-17T23:38:14.359 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_present cluster1 mirror image_6 deleted 2024-09-17T23:38:14.359 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:38:14.359 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:38:14.359 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_6 2024-09-17T23:38:14.359 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=deleted 2024-09-17T23:38:14.359 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:38:14.359 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:38:14.359 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:38:14.360 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_6 2024-09-17T23:38:14.360 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:38:14.360 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:38:14.360 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_6 2024-09-17T23:38:14.360 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-09-17T23:38:14.360 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:38:14.395 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_6: (2) No such file or directory 2024-09-17T23:38:14.400 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image_id= 2024-09-17T23:38:14.400 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-17T23:38:14.400 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.1 2024-09-17T23:38:14.502 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_present cluster1 mirror image_6 deleted '' 2024-09-17T23:38:14.502 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:38:14.502 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:38:14.502 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_6 2024-09-17T23:38:14.502 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=deleted 2024-09-17T23:38:14.502 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:38:14.502 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=deleted 2024-09-17T23:38:14.502 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_image_id 2024-09-17T23:38:14.502 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_6 2024-09-17T23:38:14.502 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:38:14.502 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:38:14.502 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_6 2024-09-17T23:38:14.503 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-09-17T23:38:14.503 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:38:14.536 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_6: (2) No such file or directory 2024-09-17T23:38:14.540 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_image_id= 2024-09-17T23:38:14.540 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:38:14.540 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test deleted = deleted 2024-09-17T23:38:14.541 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:38:14.541 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ purge_snapshots cluster2 mirror image_6 2024-09-17T23:38:14.541 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:38:14.541 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:38:14.541 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_6 2024-09-17T23:38:14.541 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap purge mirror/image_6 2024-09-17T23:38:14.587 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image_retry cluster2 mirror image_6 2024-09-17T23:38:14.587 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:38:14.587 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:38:14.588 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_6 2024-09-17T23:38:14.588 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-17T23:38:14.588 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0 2024-09-17T23:38:14.588 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image cluster2 mirror image_6 2024-09-17T23:38:14.588 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:38:14.588 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:38:14.588 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_6 2024-09-17T23:38:14.589 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_6 2024-09-17T23:38:14.632 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 rm mirror/image_6 2024-09-17T23:38:14.878 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.5102202250521612) > weight (0.25) 2024-09-17T23:38:14.878 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2024-09-17T23:38:14.879 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2024-09-17T23:38:14.879 INFO:tasks.rbd_mirror_thrash:waiting for 9 secs before reviving daemons 2024-09-17T23:38:14.879 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi038.stderr:2024-09-17T23:38:14.877+0000 7f31dd5a9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 67537) UID: 1000 2024-09-17T23:38:14.903 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-17T23:38:14.907 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:38:14.908 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 10` 2024-09-17T23:38:14.908 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_7 2024-09-17T23:38:14.908 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_image cluster2 mirror image_7 512M 2024-09-17T23:38:14.908 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:38:14.908 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:38:14.908 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:38:14.908 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:38:14.908 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_7 2024-09-17T23:38:14.908 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:38:14.908 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local size=128 2024-09-17T23:38:14.908 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n 512M ']' 2024-09-17T23:38:14.908 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ size=512M 2024-09-17T23:38:14.908 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:38:14.909 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_7 2024-09-17T23:38:14.960 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ enable_mirror cluster2 mirror image_7 2024-09-17T23:38:14.961 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:38:14.961 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:38:14.961 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_7 2024-09-17T23:38:14.961 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=snapshot 2024-09-17T23:38:14.961 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_7 snapshot 2024-09-17T23:38:15.340 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Mirroring enabled 2024-09-17T23:38:15.351 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 info mirror/image_7 2024-09-17T23:38:15.389 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'image_7': 2024-09-17T23:38:15.389 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 512 MiB in 128 objects 2024-09-17T23:38:15.389 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:38:15.389 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 1 2024-09-17T23:38:15.389 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 1a01e89fa5a7 2024-09-17T23:38:15.389 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.1a01e89fa5a7 2024-09-17T23:38:15.390 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:38:15.390 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-17T23:38:15.390 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:38:15.390 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:38:15.390 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:38:14 2024 2024-09-17T23:38:15.390 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:38:14 2024 2024-09-17T23:38:15.390 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:38:14 2024 2024-09-17T23:38:15.390 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:38:15.390 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:38:15.390 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: 07f987ed-aaea-49ff-be87-d7f4c0fc91dc 2024-09-17T23:38:15.390 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: true 2024-09-17T23:38:15.392 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ stress_write_image cluster2 mirror image_7 2024-09-17T23:38:15.392 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:38:15.393 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:38:15.393 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_7 2024-09-17T23:38:15.393 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-17T23:38:15.394 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local duration=5 2024-09-17T23:38:15.394 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set +e 2024-09-17T23:38:15.395 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_7 --debug-rbd=20 --debug-journaler=20 2024-09-17T23:38:15.617 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:100 IOs, 3697664 bytes 2024-09-17T23:38:15.776 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:200 IOs, 51999744 bytes 2024-09-17T23:38:15.881 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:300 IOs, 59059200 bytes 2024-09-17T23:38:15.982 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:400 IOs, 93136896 bytes 2024-09-17T23:38:16.092 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:500 IOs, 117447680 bytes 2024-09-17T23:38:16.187 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:600 IOs, 126117888 bytes 2024-09-17T23:38:16.273 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:700 IOs, 144855040 bytes 2024-09-17T23:38:16.365 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:800 IOs, 155046912 bytes 2024-09-17T23:38:16.464 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:900 IOs, 160231424 bytes 2024-09-17T23:38:16.549 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1000 IOs, 169540608 bytes 2024-09-17T23:38:16.677 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1100 IOs, 224360448 bytes 2024-09-17T23:38:16.830 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1200 IOs, 309004288 bytes 2024-09-17T23:38:17.015 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1300 IOs, 430843904 bytes 2024-09-17T23:38:17.101 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1400 IOs, 433982464 bytes 2024-09-17T23:38:17.239 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1500 IOs, 507174912 bytes 2024-09-17T23:38:17.383 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1600 IOs, 576450560 bytes 2024-09-17T23:38:17.473 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1700 IOs, 586109952 bytes 2024-09-17T23:38:17.558 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1800 IOs, 587741184 bytes 2024-09-17T23:38:17.666 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1900 IOs, 622215168 bytes 2024-09-17T23:38:17.756 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2000 IOs, 634322944 bytes 2024-09-17T23:38:17.855 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2100 IOs, 658162688 bytes 2024-09-17T23:38:17.979 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2200 IOs, 703226880 bytes 2024-09-17T23:38:18.107 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2300 IOs, 761142272 bytes 2024-09-17T23:38:18.191 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2400 IOs, 761768960 bytes 2024-09-17T23:38:18.329 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2500 IOs, 825180160 bytes 2024-09-17T23:38:18.431 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2600 IOs, 846914560 bytes 2024-09-17T23:38:18.516 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2700 IOs, 853422080 bytes 2024-09-17T23:38:18.605 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2800 IOs, 863813632 bytes 2024-09-17T23:38:18.758 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2900 IOs, 954694656 bytes 2024-09-17T23:38:18.893 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3000 IOs, 1004248064 bytes 2024-09-17T23:38:18.992 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3100 IOs, 1023552512 bytes 2024-09-17T23:38:19.135 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3200 IOs, 1100525568 bytes 2024-09-17T23:38:19.243 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3300 IOs, 1127237632 bytes 2024-09-17T23:38:19.372 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3400 IOs, 1187280896 bytes 2024-09-17T23:38:19.502 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3500 IOs, 1234378752 bytes 2024-09-17T23:38:19.633 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3600 IOs, 1289319424 bytes 2024-09-17T23:38:19.744 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3700 IOs, 1326362624 bytes 2024-09-17T23:38:19.847 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3800 IOs, 1348835328 bytes 2024-09-17T23:38:19.947 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3900 IOs, 1371356160 bytes 2024-09-17T23:38:20.077 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4000 IOs, 1426790400 bytes 2024-09-17T23:38:20.220 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4100 IOs, 1498172416 bytes 2024-09-17T23:38:20.368 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4200 IOs, 1568452608 bytes 2024-09-17T23:38:20.403 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ error_code=124 2024-09-17T23:38:20.404 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set -e 2024-09-17T23:38:20.404 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' 124 -eq 124 ']' 2024-09-17T23:38:20.404 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:38:20.404 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_present cluster1 mirror image_7 present 2024-09-17T23:38:20.404 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:38:20.404 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:38:20.404 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_7 2024-09-17T23:38:20.404 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=present 2024-09-17T23:38:20.404 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:38:20.404 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:38:20.404 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:38:20.405 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_7 2024-09-17T23:38:20.405 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:38:20.405 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:38:20.405 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_7 2024-09-17T23:38:20.405 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-09-17T23:38:20.406 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:38:20.450 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image_id=12b5c07c2615 2024-09-17T23:38:20.450 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-17T23:38:20.450 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.1 2024-09-17T23:38:20.551 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_present cluster1 mirror image_7 present 12b5c07c2615 2024-09-17T23:38:20.552 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:38:20.552 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:38:20.552 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_7 2024-09-17T23:38:20.552 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=present 2024-09-17T23:38:20.552 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id=12b5c07c2615 2024-09-17T23:38:20.552 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=deleted 2024-09-17T23:38:20.552 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_image_id 2024-09-17T23:38:20.552 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_7 2024-09-17T23:38:20.552 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:38:20.552 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:38:20.553 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_7 2024-09-17T23:38:20.553 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-09-17T23:38:20.553 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:38:20.594 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_image_id=12b5c07c2615 2024-09-17T23:38:20.595 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n 12b5c07c2615 2024-09-17T23:38:20.595 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -z 12b5c07c2615 -o 12b5c07c2615 = 12b5c07c2615 2024-09-17T23:38:20.595 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_state=present 2024-09-17T23:38:20.595 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test present = present 2024-09-17T23:38:20.595 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:38:20.595 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ disable_mirror cluster2 mirror image_7 2024-09-17T23:38:20.595 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:38:20.595 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:38:20.595 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_7 2024-09-17T23:38:20.595 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_7 2024-09-17T23:38:23.879 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2024-09-17T23:38:23.880 INFO:teuthology.orchestra.run:waiting for 600 2024-09-17T23:38:23.880 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2024-09-17T23:38:23.880 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2024-09-17T23:38:23.880 DEBUG:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2024-09-17T23:38:23.883 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2024-09-17T23:38:23.883 INFO:tasks.rbd_mirror_thrash:waiting for 97 secs before thrashing 2024-09-17T23:38:25.323 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi038.stderr:2024-09-17T23:38:25.321+0000 7fb609ab3640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55d744fc01a0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-17T23:38:25.324 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi038.stderr:2024-09-17T23:38:25.321+0000 7fb6102c0640 -1 rbd::mirror::ImageReplayer: 0x55d745a45900 [3/07f987ed-aaea-49ff-be87-d7f4c0fc91dc] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-17T23:38:50.256 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi038.stderr:2024-09-17T23:38:50.254+0000 7efed93cd640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55626532a000 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=191 2024-09-17T23:38:50.257 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi038.stderr:2024-09-17T23:38:50.254+0000 7efed93cd640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55626532a000 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=191, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=0ba5654d-47fb-4672-b20c-7359a1340314, primary_snap_id=bf, last_copied_object_number=128, snap_seqs={191=18446744073709551614}] 2024-09-17T23:38:50.275 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Mirroring disabled 2024-09-17T23:38:50.299 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_present cluster1 mirror image_7 deleted 2024-09-17T23:38:50.299 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:38:50.299 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:38:50.299 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_7 2024-09-17T23:38:50.299 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=deleted 2024-09-17T23:38:50.299 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:38:50.299 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:38:50.299 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:38:50.299 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_7 2024-09-17T23:38:50.300 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:38:50.300 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:38:50.300 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_7 2024-09-17T23:38:50.300 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-09-17T23:38:50.300 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:38:50.337 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_7: (2) No such file or directory 2024-09-17T23:38:50.340 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image_id= 2024-09-17T23:38:50.340 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-17T23:38:50.340 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.1 2024-09-17T23:38:50.442 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_present cluster1 mirror image_7 deleted '' 2024-09-17T23:38:50.442 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:38:50.442 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:38:50.442 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_7 2024-09-17T23:38:50.442 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=deleted 2024-09-17T23:38:50.442 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:38:50.442 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=deleted 2024-09-17T23:38:50.443 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_image_id 2024-09-17T23:38:50.443 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_7 2024-09-17T23:38:50.443 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:38:50.443 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:38:50.443 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_7 2024-09-17T23:38:50.443 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-09-17T23:38:50.443 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:38:50.476 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_7: (2) No such file or directory 2024-09-17T23:38:50.480 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_image_id= 2024-09-17T23:38:50.480 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:38:50.480 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test deleted = deleted 2024-09-17T23:38:50.480 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:38:50.480 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ purge_snapshots cluster2 mirror image_7 2024-09-17T23:38:50.480 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:38:50.480 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:38:50.480 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_7 2024-09-17T23:38:50.480 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap purge mirror/image_7 2024-09-17T23:38:50.525 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image_retry cluster2 mirror image_7 2024-09-17T23:38:50.525 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:38:50.525 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:38:50.525 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_7 2024-09-17T23:38:50.526 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-17T23:38:50.526 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0 2024-09-17T23:38:50.526 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image cluster2 mirror image_7 2024-09-17T23:38:50.526 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:38:50.526 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:38:50.526 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_7 2024-09-17T23:38:50.526 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_7 2024-09-17T23:38:50.570 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 rm mirror/image_7 2024-09-17T23:38:50.835 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-17T23:38:50.839 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:38:50.839 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 10` 2024-09-17T23:38:50.839 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_8 2024-09-17T23:38:50.839 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_image cluster2 mirror image_8 512M 2024-09-17T23:38:50.839 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:38:50.839 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:38:50.839 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:38:50.839 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:38:50.839 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_8 2024-09-17T23:38:50.839 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:38:50.839 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local size=128 2024-09-17T23:38:50.839 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n 512M ']' 2024-09-17T23:38:50.840 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ size=512M 2024-09-17T23:38:50.840 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:38:50.840 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_8 2024-09-17T23:38:50.890 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ enable_mirror cluster2 mirror image_8 2024-09-17T23:38:50.890 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:38:50.890 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:38:50.890 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_8 2024-09-17T23:38:50.890 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=snapshot 2024-09-17T23:38:50.890 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_8 snapshot 2024-09-17T23:38:51.277 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Mirroring enabled 2024-09-17T23:38:51.288 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 info mirror/image_8 2024-09-17T23:38:51.326 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'image_8': 2024-09-17T23:38:51.326 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 512 MiB in 128 objects 2024-09-17T23:38:51.327 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:38:51.327 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 1 2024-09-17T23:38:51.327 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 1a1d2946feb3 2024-09-17T23:38:51.327 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.1a1d2946feb3 2024-09-17T23:38:51.327 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:38:51.327 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-17T23:38:51.327 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:38:51.327 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:38:51.327 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:38:50 2024 2024-09-17T23:38:51.327 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:38:50 2024 2024-09-17T23:38:51.327 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:38:50 2024 2024-09-17T23:38:51.327 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:38:51.327 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:38:51.328 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: 09298336-4781-4f79-8158-83141ab4ce46 2024-09-17T23:38:51.328 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: true 2024-09-17T23:38:51.329 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ stress_write_image cluster2 mirror image_8 2024-09-17T23:38:51.330 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:38:51.330 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:38:51.330 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_8 2024-09-17T23:38:51.330 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-17T23:38:51.331 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local duration=7 2024-09-17T23:38:51.331 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set +e 2024-09-17T23:38:51.332 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_8 --debug-rbd=20 --debug-journaler=20 2024-09-17T23:38:51.591 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:100 IOs, 45979648 bytes 2024-09-17T23:38:51.773 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:200 IOs, 120717312 bytes 2024-09-17T23:38:51.918 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:300 IOs, 184449024 bytes 2024-09-17T23:38:52.051 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:400 IOs, 241320960 bytes 2024-09-17T23:38:52.159 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:500 IOs, 275836928 bytes 2024-09-17T23:38:52.285 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:600 IOs, 325019648 bytes 2024-09-17T23:38:52.398 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:700 IOs, 358507520 bytes 2024-09-17T23:38:52.482 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:800 IOs, 360260608 bytes 2024-09-17T23:38:52.617 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:900 IOs, 436389888 bytes 2024-09-17T23:38:52.720 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1000 IOs, 455640064 bytes 2024-09-17T23:38:52.826 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1100 IOs, 484047872 bytes 2024-09-17T23:38:52.949 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1200 IOs, 536943616 bytes 2024-09-17T23:38:53.051 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1300 IOs, 558617600 bytes 2024-09-17T23:38:53.168 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1400 IOs, 604914688 bytes 2024-09-17T23:38:53.312 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1500 IOs, 668653568 bytes 2024-09-17T23:38:53.433 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1600 IOs, 709493760 bytes 2024-09-17T23:38:53.512 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1700 IOs, 710102016 bytes 2024-09-17T23:38:53.634 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1800 IOs, 755713024 bytes 2024-09-17T23:38:53.779 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1900 IOs, 831029248 bytes 2024-09-17T23:38:53.937 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2000 IOs, 926464000 bytes 2024-09-17T23:38:54.062 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2100 IOs, 980400128 bytes 2024-09-17T23:38:54.211 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2200 IOs, 1030994944 bytes 2024-09-17T23:38:54.317 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2300 IOs, 1074836480 bytes 2024-09-17T23:38:54.439 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2400 IOs, 1127896064 bytes 2024-09-17T23:38:54.774 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2500 IOs, 1196442624 bytes 2024-09-17T23:38:54.892 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2600 IOs, 1250539520 bytes 2024-09-17T23:38:54.992 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2700 IOs, 1284029440 bytes 2024-09-17T23:38:55.126 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2800 IOs, 1321502720 bytes 2024-09-17T23:38:55.236 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2900 IOs, 1351000064 bytes 2024-09-17T23:38:55.325 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3000 IOs, 1358860288 bytes 2024-09-17T23:38:55.450 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3100 IOs, 1407682560 bytes 2024-09-17T23:38:55.578 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3200 IOs, 1457688576 bytes 2024-09-17T23:38:55.711 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3300 IOs, 1513364480 bytes 2024-09-17T23:38:55.853 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3400 IOs, 1589047296 bytes 2024-09-17T23:38:55.959 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3500 IOs, 1636701184 bytes 2024-09-17T23:38:56.099 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3600 IOs, 1685642240 bytes 2024-09-17T23:38:56.227 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3700 IOs, 1734906880 bytes 2024-09-17T23:38:56.360 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3800 IOs, 1790023680 bytes 2024-09-17T23:38:56.504 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3900 IOs, 1851132928 bytes 2024-09-17T23:38:56.604 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4000 IOs, 1871157248 bytes 2024-09-17T23:38:56.744 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4100 IOs, 1934195712 bytes 2024-09-17T23:38:56.868 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4200 IOs, 1976596480 bytes 2024-09-17T23:38:56.984 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4300 IOs, 2010609664 bytes 2024-09-17T23:38:57.143 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4400 IOs, 2101209088 bytes 2024-09-17T23:38:57.278 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4500 IOs, 2139892736 bytes 2024-09-17T23:38:57.404 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4600 IOs, 2188443648 bytes 2024-09-17T23:38:57.536 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4700 IOs, 2242882560 bytes 2024-09-17T23:38:57.651 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4800 IOs, 2295628800 bytes 2024-09-17T23:38:57.789 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4900 IOs, 2340590592 bytes 2024-09-17T23:38:57.920 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5000 IOs, 2404808704 bytes 2024-09-17T23:38:58.022 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5100 IOs, 2413267968 bytes 2024-09-17T23:38:58.115 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5200 IOs, 2435799040 bytes 2024-09-17T23:38:58.234 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5300 IOs, 2464232448 bytes 2024-09-17T23:38:58.338 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ error_code=124 2024-09-17T23:38:58.338 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set -e 2024-09-17T23:38:58.339 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' 124 -eq 124 ']' 2024-09-17T23:38:58.339 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:38:58.339 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_present cluster1 mirror image_8 present 2024-09-17T23:38:58.339 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:38:58.339 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:38:58.339 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_8 2024-09-17T23:38:58.339 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=present 2024-09-17T23:38:58.339 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:38:58.339 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:38:58.339 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:38:58.339 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_8 2024-09-17T23:38:58.339 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:38:58.339 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:38:58.340 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_8 2024-09-17T23:38:58.340 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-09-17T23:38:58.340 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:38:58.392 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image_id=12b56517260c 2024-09-17T23:38:58.393 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-17T23:38:58.393 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.1 2024-09-17T23:38:58.494 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_present cluster1 mirror image_8 present 12b56517260c 2024-09-17T23:38:58.495 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:38:58.495 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:38:58.495 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_8 2024-09-17T23:38:58.495 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=present 2024-09-17T23:38:58.495 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id=12b56517260c 2024-09-17T23:38:58.495 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=deleted 2024-09-17T23:38:58.495 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_image_id 2024-09-17T23:38:58.495 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_8 2024-09-17T23:38:58.495 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:38:58.495 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:38:58.495 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_8 2024-09-17T23:38:58.495 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-09-17T23:38:58.495 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:38:58.539 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_image_id=12b56517260c 2024-09-17T23:38:58.540 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n 12b56517260c 2024-09-17T23:38:58.540 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -z 12b56517260c -o 12b56517260c = 12b56517260c 2024-09-17T23:38:58.540 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_state=present 2024-09-17T23:38:58.540 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test present = present 2024-09-17T23:38:58.540 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:38:58.540 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ disable_mirror cluster2 mirror image_8 2024-09-17T23:38:58.540 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:38:58.540 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:38:58.540 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_8 2024-09-17T23:38:58.540 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_8 2024-09-17T23:39:04.998 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-09-17T23:39:04.999 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-09-17T23:39:04.999 INFO:tasks.rbd_mirror_thrash:waiting for 2 secs before reviving daemons 2024-09-17T23:39:05.000 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi038.stderr:2024-09-17T23:39:04.998+0000 7f8368ab7640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 62927) UID: 1000 2024-09-17T23:39:07.000 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-09-17T23:39:07.001 INFO:teuthology.orchestra.run:waiting for 600 2024-09-17T23:39:07.001 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-09-17T23:39:07.001 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-09-17T23:39:07.001 DEBUG:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-09-17T23:39:07.005 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-09-17T23:39:07.005 INFO:tasks.rbd_mirror_thrash:waiting for 102 secs before thrashing 2024-09-17T23:39:25.324 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi038.stderr:2024-09-17T23:39:25.321+0000 7fb609ab3640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55d744fc11e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-17T23:39:25.324 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi038.stderr:2024-09-17T23:39:25.321+0000 7fb60fabf640 -1 rbd::mirror::ImageReplayer: 0x55d745a44a00 [3/09298336-4781-4f79-8158-83141ab4ce46] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-17T23:39:29.373 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Mirroring disabled 2024-09-17T23:39:29.399 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_present cluster1 mirror image_8 deleted 2024-09-17T23:39:29.399 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:39:29.399 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:39:29.399 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_8 2024-09-17T23:39:29.399 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=deleted 2024-09-17T23:39:29.399 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:39:29.399 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:39:29.399 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:39:29.399 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_8 2024-09-17T23:39:29.399 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:39:29.399 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:39:29.399 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_8 2024-09-17T23:39:29.400 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-09-17T23:39:29.400 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:39:29.438 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_8: (2) No such file or directory 2024-09-17T23:39:29.442 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image_id= 2024-09-17T23:39:29.442 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-17T23:39:29.442 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.1 2024-09-17T23:39:29.544 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_present cluster1 mirror image_8 deleted '' 2024-09-17T23:39:29.544 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:39:29.544 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:39:29.544 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_8 2024-09-17T23:39:29.545 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=deleted 2024-09-17T23:39:29.545 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:39:29.545 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=deleted 2024-09-17T23:39:29.545 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_image_id 2024-09-17T23:39:29.545 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_8 2024-09-17T23:39:29.545 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:39:29.545 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:39:29.545 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_8 2024-09-17T23:39:29.545 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-09-17T23:39:29.545 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:39:29.581 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_8: (2) No such file or directory 2024-09-17T23:39:29.584 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_image_id= 2024-09-17T23:39:29.584 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:39:29.584 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test deleted = deleted 2024-09-17T23:39:29.585 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:39:29.585 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ purge_snapshots cluster2 mirror image_8 2024-09-17T23:39:29.585 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:39:29.585 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:39:29.585 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_8 2024-09-17T23:39:29.585 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap purge mirror/image_8 2024-09-17T23:39:29.630 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image_retry cluster2 mirror image_8 2024-09-17T23:39:29.630 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:39:29.631 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:39:29.631 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_8 2024-09-17T23:39:29.631 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-17T23:39:29.631 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0 2024-09-17T23:39:29.632 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image cluster2 mirror image_8 2024-09-17T23:39:29.632 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:39:29.632 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:39:29.632 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_8 2024-09-17T23:39:29.632 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_8 2024-09-17T23:39:29.676 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 rm mirror/image_8 2024-09-17T23:39:29.939 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-17T23:39:29.943 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:39:29.943 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 10` 2024-09-17T23:39:29.943 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_9 2024-09-17T23:39:29.943 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_image cluster2 mirror image_9 512M 2024-09-17T23:39:29.943 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:39:29.943 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:39:29.943 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:39:29.943 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:39:29.943 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_9 2024-09-17T23:39:29.943 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:39:29.943 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local size=128 2024-09-17T23:39:29.944 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n 512M ']' 2024-09-17T23:39:29.944 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ size=512M 2024-09-17T23:39:29.944 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:39:29.944 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_9 2024-09-17T23:39:29.995 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ enable_mirror cluster2 mirror image_9 2024-09-17T23:39:29.995 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:39:29.995 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:39:29.995 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_9 2024-09-17T23:39:29.995 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=snapshot 2024-09-17T23:39:29.995 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_9 snapshot 2024-09-17T23:39:30.374 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Mirroring enabled 2024-09-17T23:39:30.384 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 info mirror/image_9 2024-09-17T23:39:30.422 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'image_9': 2024-09-17T23:39:30.422 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 512 MiB in 128 objects 2024-09-17T23:39:30.422 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:39:30.422 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 1 2024-09-17T23:39:30.422 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 1a38136f0f79 2024-09-17T23:39:30.423 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.1a38136f0f79 2024-09-17T23:39:30.423 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:39:30.423 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-17T23:39:30.423 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:39:30.423 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:39:30.423 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:39:29 2024 2024-09-17T23:39:30.423 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:39:29 2024 2024-09-17T23:39:30.423 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:39:29 2024 2024-09-17T23:39:30.423 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:39:30.423 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:39:30.423 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: 6f16d14d-f972-43d9-abf6-98dc223adc43 2024-09-17T23:39:30.423 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: true 2024-09-17T23:39:30.426 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ stress_write_image cluster2 mirror image_9 2024-09-17T23:39:30.426 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:39:30.426 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:39:30.426 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_9 2024-09-17T23:39:30.426 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-17T23:39:30.428 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local duration=6 2024-09-17T23:39:30.428 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set +e 2024-09-17T23:39:30.428 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_9 --debug-rbd=20 --debug-journaler=20 2024-09-17T23:39:30.679 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:100 IOs, 23431168 bytes 2024-09-17T23:39:30.853 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:200 IOs, 91770880 bytes 2024-09-17T23:39:30.967 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:300 IOs, 124390400 bytes 2024-09-17T23:39:31.069 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:400 IOs, 140798976 bytes 2024-09-17T23:39:31.159 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:500 IOs, 148257792 bytes 2024-09-17T23:39:31.284 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:600 IOs, 199690240 bytes 2024-09-17T23:39:31.391 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:700 IOs, 221703168 bytes 2024-09-17T23:39:31.507 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:800 IOs, 256910336 bytes 2024-09-17T23:39:31.611 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:900 IOs, 282694656 bytes 2024-09-17T23:39:31.709 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1000 IOs, 301404160 bytes 2024-09-17T23:39:31.822 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1100 IOs, 342528000 bytes 2024-09-17T23:39:31.957 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1200 IOs, 406005760 bytes 2024-09-17T23:39:32.042 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1300 IOs, 414070784 bytes 2024-09-17T23:39:32.154 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1400 IOs, 448850944 bytes 2024-09-17T23:39:32.275 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1500 IOs, 487863296 bytes 2024-09-17T23:39:32.391 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1600 IOs, 524843008 bytes 2024-09-17T23:39:32.525 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1700 IOs, 588582912 bytes 2024-09-17T23:39:32.636 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1800 IOs, 620331008 bytes 2024-09-17T23:39:32.785 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1900 IOs, 702148608 bytes 2024-09-17T23:39:32.890 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2000 IOs, 723539968 bytes 2024-09-17T23:39:33.002 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2100 IOs, 752800768 bytes 2024-09-17T23:39:33.110 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2200 IOs, 789799936 bytes 2024-09-17T23:39:33.208 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2300 IOs, 799187968 bytes 2024-09-17T23:39:33.331 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2400 IOs, 847051776 bytes 2024-09-17T23:39:33.437 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2500 IOs, 879147008 bytes 2024-09-17T23:39:33.544 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2600 IOs, 901344256 bytes 2024-09-17T23:39:33.661 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2700 IOs, 941420544 bytes 2024-09-17T23:39:33.782 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2800 IOs, 993879040 bytes 2024-09-17T23:39:33.924 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2900 IOs, 1050887168 bytes 2024-09-17T23:39:34.128 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3000 IOs, 1192691712 bytes 2024-09-17T23:39:34.219 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3100 IOs, 1197868032 bytes 2024-09-17T23:39:34.364 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3200 IOs, 1263071232 bytes 2024-09-17T23:39:34.538 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3300 IOs, 1364564992 bytes 2024-09-17T23:39:34.669 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3400 IOs, 1420963840 bytes 2024-09-17T23:39:34.793 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3500 IOs, 1478548480 bytes 2024-09-17T23:39:34.935 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3600 IOs, 1541940224 bytes 2024-09-17T23:39:35.032 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3700 IOs, 1555059712 bytes 2024-09-17T23:39:35.133 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3800 IOs, 1570295808 bytes 2024-09-17T23:39:35.236 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3900 IOs, 1591442432 bytes 2024-09-17T23:39:35.325 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4000 IOs, 1592056832 bytes 2024-09-17T23:39:35.656 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4100 IOs, 1645822976 bytes 2024-09-17T23:39:35.787 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4200 IOs, 1701071872 bytes 2024-09-17T23:39:35.927 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4300 IOs, 1766838272 bytes 2024-09-17T23:39:36.034 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4400 IOs, 1794723840 bytes 2024-09-17T23:39:36.153 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4500 IOs, 1835510784 bytes 2024-09-17T23:39:36.263 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4600 IOs, 1864701952 bytes 2024-09-17T23:39:36.372 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4700 IOs, 1892163584 bytes 2024-09-17T23:39:36.435 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ error_code=124 2024-09-17T23:39:36.435 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set -e 2024-09-17T23:39:36.435 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' 124 -eq 124 ']' 2024-09-17T23:39:36.436 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:39:36.436 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_present cluster1 mirror image_9 present 2024-09-17T23:39:36.436 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:39:36.436 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:39:36.436 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_9 2024-09-17T23:39:36.436 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=present 2024-09-17T23:39:36.436 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:39:36.436 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:39:36.437 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:39:36.437 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_9 2024-09-17T23:39:36.437 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:39:36.437 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:39:36.437 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_9 2024-09-17T23:39:36.438 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-09-17T23:39:36.438 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:39:36.484 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image_id=12b5d12cc12b 2024-09-17T23:39:36.484 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-17T23:39:36.484 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.1 2024-09-17T23:39:36.585 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_present cluster1 mirror image_9 present 12b5d12cc12b 2024-09-17T23:39:36.586 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:39:36.586 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:39:36.586 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_9 2024-09-17T23:39:36.586 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=present 2024-09-17T23:39:36.586 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id=12b5d12cc12b 2024-09-17T23:39:36.586 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=deleted 2024-09-17T23:39:36.586 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_image_id 2024-09-17T23:39:36.586 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_9 2024-09-17T23:39:36.586 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:39:36.586 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:39:36.586 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_9 2024-09-17T23:39:36.586 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-09-17T23:39:36.586 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:39:36.631 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_image_id=12b5d12cc12b 2024-09-17T23:39:36.632 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n 12b5d12cc12b 2024-09-17T23:39:36.632 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -z 12b5d12cc12b -o 12b5d12cc12b = 12b5d12cc12b 2024-09-17T23:39:36.632 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_state=present 2024-09-17T23:39:36.632 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test present = present 2024-09-17T23:39:36.632 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:39:36.632 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ disable_mirror cluster2 mirror image_9 2024-09-17T23:39:36.632 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:39:36.632 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:39:36.632 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_9 2024-09-17T23:39:36.632 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_9 2024-09-17T23:39:55.324 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi038.stderr:2024-09-17T23:39:55.322+0000 7fb6092b2640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55d744fc1520 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-17T23:39:55.325 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi038.stderr:2024-09-17T23:39:55.322+0000 7fb60fabf640 -1 rbd::mirror::ImageReplayer: 0x55d7450c2000 [3/6f16d14d-f972-43d9-abf6-98dc223adc43] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-17T23:40:00.884 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.7023334744412562) > weight (0.25) 2024-09-17T23:40:00.885 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2024-09-17T23:40:00.885 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2024-09-17T23:40:00.885 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2024-09-17T23:40:00.886 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi038.stderr:2024-09-17T23:40:00.884+0000 7f128aa30640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 69945) UID: 1000 2024-09-17T23:40:07.886 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2024-09-17T23:40:07.886 INFO:teuthology.orchestra.run:waiting for 600 2024-09-17T23:40:07.887 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2024-09-17T23:40:07.887 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2024-09-17T23:40:07.887 DEBUG:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2024-09-17T23:40:07.890 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2024-09-17T23:40:07.891 INFO:tasks.rbd_mirror_thrash:waiting for 92 secs before thrashing 2024-09-17T23:40:08.188 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Mirroring disabled 2024-09-17T23:40:08.215 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_present cluster1 mirror image_9 deleted 2024-09-17T23:40:08.215 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:40:08.215 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:40:08.215 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_9 2024-09-17T23:40:08.215 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=deleted 2024-09-17T23:40:08.215 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:40:08.215 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:40:08.216 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:40:08.216 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_9 2024-09-17T23:40:08.216 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:40:08.216 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:40:08.216 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_9 2024-09-17T23:40:08.216 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-09-17T23:40:08.216 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:40:08.250 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_9: (2) No such file or directory 2024-09-17T23:40:08.253 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image_id= 2024-09-17T23:40:08.254 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-17T23:40:08.254 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.1 2024-09-17T23:40:08.355 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_present cluster1 mirror image_9 deleted '' 2024-09-17T23:40:08.355 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:40:08.355 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:40:08.356 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_9 2024-09-17T23:40:08.356 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=deleted 2024-09-17T23:40:08.356 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:40:08.356 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=deleted 2024-09-17T23:40:08.356 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_image_id 2024-09-17T23:40:08.356 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_9 2024-09-17T23:40:08.356 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:40:08.356 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:40:08.356 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_9 2024-09-17T23:40:08.356 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-09-17T23:40:08.356 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:40:08.396 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_9: (2) No such file or directory 2024-09-17T23:40:08.400 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_image_id= 2024-09-17T23:40:08.400 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:40:08.400 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test deleted = deleted 2024-09-17T23:40:08.400 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:40:08.400 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ purge_snapshots cluster2 mirror image_9 2024-09-17T23:40:08.400 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:40:08.400 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:40:08.400 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_9 2024-09-17T23:40:08.400 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap purge mirror/image_9 2024-09-17T23:40:08.445 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image_retry cluster2 mirror image_9 2024-09-17T23:40:08.445 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:40:08.445 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:40:08.445 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_9 2024-09-17T23:40:08.445 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-17T23:40:08.445 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0 2024-09-17T23:40:08.446 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image cluster2 mirror image_9 2024-09-17T23:40:08.446 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:40:08.446 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:40:08.447 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_9 2024-09-17T23:40:08.447 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_9 2024-09-17T23:40:08.491 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 rm mirror/image_9 2024-09-17T23:40:08.756 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-17T23:40:08.760 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:40:08.760 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for i in `seq 1 10` 2024-09-17T23:40:08.760 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image=image_10 2024-09-17T23:40:08.761 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ create_image cluster2 mirror image_10 512M 2024-09-17T23:40:08.761 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:40:08.761 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:40:08.761 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:40:08.761 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:40:08.761 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_10 2024-09-17T23:40:08.761 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:40:08.761 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local size=128 2024-09-17T23:40:08.761 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -n 512M ']' 2024-09-17T23:40:08.761 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ size=512M 2024-09-17T23:40:08.761 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ shift 2024-09-17T23:40:08.762 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_10 2024-09-17T23:40:08.812 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ enable_mirror cluster2 mirror image_10 2024-09-17T23:40:08.812 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:40:08.812 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:40:08.812 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_10 2024-09-17T23:40:08.812 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local mode=snapshot 2024-09-17T23:40:08.813 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_10 snapshot 2024-09-17T23:40:09.204 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Mirroring enabled 2024-09-17T23:40:09.215 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 info mirror/image_10 2024-09-17T23:40:09.252 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:rbd image 'image_10': 2024-09-17T23:40:09.252 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: size 512 MiB in 128 objects 2024-09-17T23:40:09.252 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: order 22 (4 MiB objects) 2024-09-17T23:40:09.252 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: snapshot_count: 1 2024-09-17T23:40:09.252 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: id: 1a512064275f 2024-09-17T23:40:09.252 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: block_name_prefix: rbd_data.1a512064275f 2024-09-17T23:40:09.253 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: format: 2 2024-09-17T23:40:09.253 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-17T23:40:09.253 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: op_features: 2024-09-17T23:40:09.253 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: flags: 2024-09-17T23:40:09.253 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: create_timestamp: Tue Sep 17 23:40:08 2024 2024-09-17T23:40:09.253 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: access_timestamp: Tue Sep 17 23:40:08 2024 2024-09-17T23:40:09.253 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: modify_timestamp: Tue Sep 17 23:40:08 2024 2024-09-17T23:40:09.253 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring state: enabled 2024-09-17T23:40:09.253 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring mode: snapshot 2024-09-17T23:40:09.253 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring global id: 03a4f4e7-2dca-4b8e-9bfe-b487b1be693f 2024-09-17T23:40:09.253 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout: mirroring primary: true 2024-09-17T23:40:09.256 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ stress_write_image cluster2 mirror image_10 2024-09-17T23:40:09.256 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:40:09.256 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:40:09.256 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_10 2024-09-17T23:40:09.256 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-17T23:40:09.257 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local duration=8 2024-09-17T23:40:09.258 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set +e 2024-09-17T23:40:09.258 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_10 --debug-rbd=20 --debug-journaler=20 2024-09-17T23:40:09.482 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:100 IOs, 5017600 bytes 2024-09-17T23:40:09.625 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:200 IOs, 38263808 bytes 2024-09-17T23:40:09.732 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:300 IOs, 59966464 bytes 2024-09-17T23:40:09.854 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:400 IOs, 94393344 bytes 2024-09-17T23:40:09.977 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:500 IOs, 159611904 bytes 2024-09-17T23:40:10.092 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:600 IOs, 182960128 bytes 2024-09-17T23:40:10.238 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:700 IOs, 267559936 bytes 2024-09-17T23:40:10.336 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:800 IOs, 272166912 bytes 2024-09-17T23:40:10.438 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:900 IOs, 294236160 bytes 2024-09-17T23:40:10.537 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1000 IOs, 316893184 bytes 2024-09-17T23:40:10.686 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1100 IOs, 397537280 bytes 2024-09-17T23:40:10.801 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1200 IOs, 435531776 bytes 2024-09-17T23:40:10.906 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1300 IOs, 466463744 bytes 2024-09-17T23:40:11.042 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1400 IOs, 527022080 bytes 2024-09-17T23:40:11.198 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1500 IOs, 615614464 bytes 2024-09-17T23:40:11.340 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1600 IOs, 714991616 bytes 2024-09-17T23:40:11.499 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1700 IOs, 768155648 bytes 2024-09-17T23:40:11.597 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1800 IOs, 785876992 bytes 2024-09-17T23:40:11.708 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:1900 IOs, 818964480 bytes 2024-09-17T23:40:11.794 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2000 IOs, 825629696 bytes 2024-09-17T23:40:11.879 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2100 IOs, 843166720 bytes 2024-09-17T23:40:12.046 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2200 IOs, 925946880 bytes 2024-09-17T23:40:12.145 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2300 IOs, 943947776 bytes 2024-09-17T23:40:12.252 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2400 IOs, 972385280 bytes 2024-09-17T23:40:12.361 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2500 IOs, 998872064 bytes 2024-09-17T23:40:12.475 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2600 IOs, 1036224512 bytes 2024-09-17T23:40:12.591 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2700 IOs, 1075267584 bytes 2024-09-17T23:40:12.681 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2800 IOs, 1095627776 bytes 2024-09-17T23:40:12.775 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:2900 IOs, 1100659712 bytes 2024-09-17T23:40:12.903 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3000 IOs, 1151519744 bytes 2024-09-17T23:40:13.018 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3100 IOs, 1191168000 bytes 2024-09-17T23:40:13.164 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3200 IOs, 1256427520 bytes 2024-09-17T23:40:13.258 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3300 IOs, 1268943872 bytes 2024-09-17T23:40:13.374 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3400 IOs, 1309119488 bytes 2024-09-17T23:40:13.497 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3500 IOs, 1355493376 bytes 2024-09-17T23:40:13.671 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3600 IOs, 1451095040 bytes 2024-09-17T23:40:13.803 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3700 IOs, 1509246976 bytes 2024-09-17T23:40:13.962 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3800 IOs, 1594948608 bytes 2024-09-17T23:40:14.054 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:3900 IOs, 1602137088 bytes 2024-09-17T23:40:14.181 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4000 IOs, 1651462144 bytes 2024-09-17T23:40:14.271 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4100 IOs, 1657515008 bytes 2024-09-17T23:40:14.379 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4200 IOs, 1685922816 bytes 2024-09-17T23:40:14.507 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4300 IOs, 1735434240 bytes 2024-09-17T23:40:14.667 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4400 IOs, 1822232576 bytes 2024-09-17T23:40:14.782 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4500 IOs, 1856558080 bytes 2024-09-17T23:40:14.926 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4600 IOs, 1922494464 bytes 2024-09-17T23:40:15.044 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4700 IOs, 1959027712 bytes 2024-09-17T23:40:15.168 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4800 IOs, 2024499200 bytes 2024-09-17T23:40:15.303 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:4900 IOs, 2063798272 bytes 2024-09-17T23:40:15.421 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5000 IOs, 2108554240 bytes 2024-09-17T23:40:15.544 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5100 IOs, 2154752000 bytes 2024-09-17T23:40:15.665 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5200 IOs, 2201402368 bytes 2024-09-17T23:40:15.770 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5300 IOs, 2229226496 bytes 2024-09-17T23:40:15.890 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5400 IOs, 2286406656 bytes 2024-09-17T23:40:15.988 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5500 IOs, 2290471936 bytes 2024-09-17T23:40:16.131 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5600 IOs, 2358873088 bytes 2024-09-17T23:40:16.256 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5700 IOs, 2402217984 bytes 2024-09-17T23:40:16.365 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5800 IOs, 2433795072 bytes 2024-09-17T23:40:16.487 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:5900 IOs, 2477583360 bytes 2024-09-17T23:40:16.571 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6000 IOs, 2478196736 bytes 2024-09-17T23:40:16.715 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6100 IOs, 2544818176 bytes 2024-09-17T23:40:16.838 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6200 IOs, 2591069184 bytes 2024-09-17T23:40:16.939 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6300 IOs, 2609650688 bytes 2024-09-17T23:40:17.059 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6400 IOs, 2656001024 bytes 2024-09-17T23:40:17.167 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:6500 IOs, 2683294720 bytes 2024-09-17T23:40:17.273 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ error_code=124 2024-09-17T23:40:17.273 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set -e 2024-09-17T23:40:17.273 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' 124 -eq 124 ']' 2024-09-17T23:40:17.274 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:40:17.274 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_present cluster1 mirror image_10 present 2024-09-17T23:40:17.274 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:40:17.274 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:40:17.274 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_10 2024-09-17T23:40:17.274 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=present 2024-09-17T23:40:17.274 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:40:17.274 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:40:17.274 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:40:17.274 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_10 2024-09-17T23:40:17.274 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:40:17.274 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:40:17.274 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_10 2024-09-17T23:40:17.275 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-09-17T23:40:17.275 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:40:17.324 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image_id=12b5fff97c7e 2024-09-17T23:40:17.324 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-17T23:40:17.324 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.1 2024-09-17T23:40:17.426 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_present cluster1 mirror image_10 present 12b5fff97c7e 2024-09-17T23:40:17.426 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:40:17.426 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:40:17.427 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_10 2024-09-17T23:40:17.427 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=present 2024-09-17T23:40:17.427 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id=12b5fff97c7e 2024-09-17T23:40:17.427 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=deleted 2024-09-17T23:40:17.427 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_image_id 2024-09-17T23:40:17.427 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_10 2024-09-17T23:40:17.427 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:40:17.428 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:40:17.428 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_10 2024-09-17T23:40:17.428 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-09-17T23:40:17.428 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:40:17.474 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_image_id=12b5fff97c7e 2024-09-17T23:40:17.474 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n 12b5fff97c7e 2024-09-17T23:40:17.474 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -z 12b5fff97c7e -o 12b5fff97c7e = 12b5fff97c7e 2024-09-17T23:40:17.474 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_state=present 2024-09-17T23:40:17.474 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test present = present 2024-09-17T23:40:17.474 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:40:17.474 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ disable_mirror cluster2 mirror image_10 2024-09-17T23:40:17.474 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:40:17.474 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:40:17.474 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_10 2024-09-17T23:40:17.474 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_10 2024-09-17T23:40:25.324 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi038.stderr:2024-09-17T23:40:25.322+0000 7fb6092b2640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55d74519d380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-17T23:40:25.324 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi038.stderr:2024-09-17T23:40:25.322+0000 7fb60fabf640 -1 rbd::mirror::ImageReplayer: 0x55d7450c2000 [3/03a4f4e7-2dca-4b8e-9bfe-b487b1be693f] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-17T23:40:48.352 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi038.stderr:2024-09-17T23:40:48.349+0000 7efed9bce640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x556268a40000 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=194 2024-09-17T23:40:48.352 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi038.stderr:2024-09-17T23:40:48.349+0000 7efed9bce640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x556268a40000 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=194, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=0ba5654d-47fb-4672-b20c-7359a1340314, primary_snap_id=c2, last_copied_object_number=128, snap_seqs={194=18446744073709551614}] 2024-09-17T23:40:48.367 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:Mirroring disabled 2024-09-17T23:40:48.397 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_present cluster1 mirror image_10 deleted 2024-09-17T23:40:48.397 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:40:48.397 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:40:48.397 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_10 2024-09-17T23:40:48.398 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local state=deleted 2024-09-17T23:40:48.398 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:40:48.398 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local s 2024-09-17T23:40:48.398 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:40:48.398 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_10 2024-09-17T23:40:48.398 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:40:48.398 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:40:48.398 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_10 2024-09-17T23:40:48.399 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-09-17T23:40:48.399 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:40:48.436 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_10: (2) No such file or directory 2024-09-17T23:40:48.439 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ image_id= 2024-09-17T23:40:48.439 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-17T23:40:48.439 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0.1 2024-09-17T23:40:48.541 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test_image_present cluster1 mirror image_10 deleted '' 2024-09-17T23:40:48.542 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:40:48.542 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:40:48.542 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_10 2024-09-17T23:40:48.542 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local test_state=deleted 2024-09-17T23:40:48.542 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image_id= 2024-09-17T23:40:48.542 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_state=deleted 2024-09-17T23:40:48.542 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local current_image_id 2024-09-17T23:40:48.542 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ get_image_id cluster1 mirror image_10 2024-09-17T23:40:48.542 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:40:48.542 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:40:48.542 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local image=image_10 2024-09-17T23:40:48.542 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-09-17T23:40:48.543 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-17T23:40:48.578 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:rbd: error opening image image_10: (2) No such file or directory 2024-09-17T23:40:48.581 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ current_image_id= 2024-09-17T23:40:48.582 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n '' 2024-09-17T23:40:48.582 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test deleted = deleted 2024-09-17T23:40:48.582 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:40:48.582 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ purge_snapshots cluster2 mirror image_10 2024-09-17T23:40:48.582 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:40:48.582 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:40:48.582 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_10 2024-09-17T23:40:48.582 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster cluster2 snap purge mirror/image_10 2024-09-17T23:40:48.627 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image_retry cluster2 mirror image_10 2024-09-17T23:40:48.627 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:40:48.627 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:40:48.627 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_10 2024-09-17T23:40:48.627 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-17T23:40:48.628 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0 2024-09-17T23:40:48.628 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ remove_image cluster2 mirror image_10 2024-09-17T23:40:48.628 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:40:48.628 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:40:48.628 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local image=image_10 2024-09-17T23:40:48.628 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_10 2024-09-17T23:40:48.673 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rbd --cluster=cluster2 rm mirror/image_10 2024-09-17T23:40:48.938 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-17T23:40:48.942 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:40:48.942 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ testlog 'TEST: check if removed images'\'' OMAP are removed' 2024-09-17T23:40:48.942 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ tee -a /tmp/tmp.s5MEFDgsej/rbd-mirror.test.log 2024-09-17T23:40:48.942 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ date '+%F %T' 2024-09-17T23:40:48.943 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ echo 2024-09-17 23:40:48 TEST: check if removed 'images'\''' OMAP are removed 2024-09-17T23:40:48.943 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:2024-09-17 23:40:48 TEST: check if removed images' OMAP are removed 2024-09-17T23:40:48.952 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_in_omap cluster1 mirror 2024-09-17T23:40:48.952 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:40:48.952 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:40:48.952 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirroring status_global 2024-09-17T23:40:48.952 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:40:48.952 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:40:48.952 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local obj_name=rbd_mirroring 2024-09-17T23:40:48.952 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local filter=status_global 2024-09-17T23:40:48.953 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-17T23:40:48.953 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0 2024-09-17T23:40:48.954 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set +e 2024-09-17T23:40:48.954 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirroring status_global 2024-09-17T23:40:48.954 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:40:48.954 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:40:48.954 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local obj_name=rbd_mirroring 2024-09-17T23:40:48.954 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local filter=status_global 2024-09-17T23:40:48.954 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ list_omap_keys cluster1 mirror rbd_mirroring 2024-09-17T23:40:48.954 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:40:48.954 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:40:48.954 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local obj_name=rbd_mirroring 2024-09-17T23:40:48.954 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirroring 2024-09-17T23:40:48.955 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ grep -c status_global 2024-09-17T23:40:49.006 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.27635821252882853) > weight (0.25) 2024-09-17T23:40:49.006 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2024-09-17T23:40:49.007 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2024-09-17T23:40:49.007 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2024-09-17T23:40:49.007 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi038.stderr:2024-09-17T23:40:49.006+0000 7fb610ac1640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 67431) UID: 1000 2024-09-17T23:40:49.053 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 0 = 0 2024-09-17T23:40:49.053 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ error_code=0 2024-09-17T23:40:49.054 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set -e 2024-09-17T23:40:49.054 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' 0 -eq 0 ']' 2024-09-17T23:40:49.054 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:40:49.054 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirroring image_ 2024-09-17T23:40:49.054 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:40:49.054 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:40:49.054 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local obj_name=rbd_mirroring 2024-09-17T23:40:49.054 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local filter=image_ 2024-09-17T23:40:49.054 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-17T23:40:49.054 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0 2024-09-17T23:40:49.054 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set +e 2024-09-17T23:40:49.054 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirroring image_ 2024-09-17T23:40:49.054 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:40:49.054 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:40:49.055 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local obj_name=rbd_mirroring 2024-09-17T23:40:49.055 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local filter=image_ 2024-09-17T23:40:49.055 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ list_omap_keys cluster1 mirror rbd_mirroring 2024-09-17T23:40:49.055 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:40:49.055 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:40:49.055 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local obj_name=rbd_mirroring 2024-09-17T23:40:49.055 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ grep -c image_ 2024-09-17T23:40:49.055 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirroring 2024-09-17T23:40:49.100 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 0 = 0 2024-09-17T23:40:49.100 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ error_code=0 2024-09-17T23:40:49.100 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set -e 2024-09-17T23:40:49.100 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' 0 -eq 0 ']' 2024-09-17T23:40:49.100 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:40:49.100 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirror_leader image_map 2024-09-17T23:40:49.100 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:40:49.100 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:40:49.100 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local obj_name=rbd_mirror_leader 2024-09-17T23:40:49.100 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local filter=image_map 2024-09-17T23:40:49.101 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-17T23:40:49.101 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0 2024-09-17T23:40:49.101 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set +e 2024-09-17T23:40:49.102 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirror_leader image_map 2024-09-17T23:40:49.102 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:40:49.102 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:40:49.102 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local obj_name=rbd_mirror_leader 2024-09-17T23:40:49.102 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local filter=image_map 2024-09-17T23:40:49.102 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ list_omap_keys cluster1 mirror rbd_mirror_leader 2024-09-17T23:40:49.102 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:40:49.102 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:40:49.102 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local obj_name=rbd_mirror_leader 2024-09-17T23:40:49.102 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirror_leader 2024-09-17T23:40:49.102 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ grep -c image_map 2024-09-17T23:40:49.136 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 1 = 0 2024-09-17T23:40:49.136 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ error_code=1 2024-09-17T23:40:49.136 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set -e 2024-09-17T23:40:49.136 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' 1 -eq 0 ']' 2024-09-17T23:40:49.136 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-17T23:40:49.136 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 1 2024-09-17T23:40:50.007 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2024-09-17T23:40:50.007 INFO:teuthology.orchestra.run:waiting for 600 2024-09-17T23:40:50.008 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2024-09-17T23:40:50.008 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2024-09-17T23:40:50.008 DEBUG:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2024-09-17T23:40:50.011 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2024-09-17T23:40:50.011 INFO:tasks.rbd_mirror_thrash:waiting for 96 secs before thrashing 2024-09-17T23:40:50.138 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set +e 2024-09-17T23:40:50.138 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirror_leader image_map 2024-09-17T23:40:50.138 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:40:50.138 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:40:50.138 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local obj_name=rbd_mirror_leader 2024-09-17T23:40:50.138 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local filter=image_map 2024-09-17T23:40:50.138 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ list_omap_keys cluster1 mirror rbd_mirror_leader 2024-09-17T23:40:50.138 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:40:50.138 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ grep -c image_map 2024-09-17T23:40:50.138 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:40:50.138 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local obj_name=rbd_mirror_leader 2024-09-17T23:40:50.138 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirror_leader 2024-09-17T23:40:50.185 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 1 = 0 2024-09-17T23:40:50.186 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ error_code=1 2024-09-17T23:40:50.186 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set -e 2024-09-17T23:40:50.186 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' 1 -eq 0 ']' 2024-09-17T23:40:50.186 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-17T23:40:50.186 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 2 2024-09-17T23:40:52.187 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set +e 2024-09-17T23:40:52.188 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirror_leader image_map 2024-09-17T23:40:52.188 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:40:52.188 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:40:52.188 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local obj_name=rbd_mirror_leader 2024-09-17T23:40:52.188 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local filter=image_map 2024-09-17T23:40:52.189 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ list_omap_keys cluster1 mirror rbd_mirror_leader 2024-09-17T23:40:52.189 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster1 2024-09-17T23:40:52.189 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:40:52.189 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local obj_name=rbd_mirror_leader 2024-09-17T23:40:52.189 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirror_leader 2024-09-17T23:40:52.189 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ grep -c image_map 2024-09-17T23:40:52.230 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 0 = 0 2024-09-17T23:40:52.230 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ error_code=0 2024-09-17T23:40:52.230 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set -e 2024-09-17T23:40:52.230 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' 0 -eq 0 ']' 2024-09-17T23:40:52.230 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:40:52.230 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_image_in_omap cluster2 mirror 2024-09-17T23:40:52.230 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:40:52.230 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:40:52.230 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirroring status_global 2024-09-17T23:40:52.230 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:40:52.230 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:40:52.231 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local obj_name=rbd_mirroring 2024-09-17T23:40:52.231 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local filter=status_global 2024-09-17T23:40:52.231 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-17T23:40:52.231 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0 2024-09-17T23:40:52.232 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set +e 2024-09-17T23:40:52.232 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ count_omap_keys_with_filter cluster2 mirror rbd_mirroring status_global 2024-09-17T23:40:52.232 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster2 2024-09-17T23:40:52.232 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:40:52.232 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local obj_name=rbd_mirroring 2024-09-17T23:40:52.232 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local filter=status_global 2024-09-17T23:40:52.232 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ list_omap_keys cluster2 mirror rbd_mirroring 2024-09-17T23:40:52.232 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster2 2024-09-17T23:40:52.232 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:40:52.232 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local obj_name=rbd_mirroring 2024-09-17T23:40:52.233 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rados --cluster cluster2 -p mirror listomapkeys rbd_mirroring 2024-09-17T23:40:52.233 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ grep -c status_global 2024-09-17T23:40:52.270 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 0 = 0 2024-09-17T23:40:52.270 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ error_code=0 2024-09-17T23:40:52.270 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set -e 2024-09-17T23:40:52.270 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' 0 -eq 0 ']' 2024-09-17T23:40:52.270 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:40:52.270 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirroring image_ 2024-09-17T23:40:52.270 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:40:52.271 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:40:52.271 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local obj_name=rbd_mirroring 2024-09-17T23:40:52.271 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local filter=image_ 2024-09-17T23:40:52.271 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-17T23:40:52.271 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0 2024-09-17T23:40:52.272 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set +e 2024-09-17T23:40:52.272 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ count_omap_keys_with_filter cluster2 mirror rbd_mirroring image_ 2024-09-17T23:40:52.272 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster2 2024-09-17T23:40:52.272 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:40:52.272 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local obj_name=rbd_mirroring 2024-09-17T23:40:52.272 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local filter=image_ 2024-09-17T23:40:52.273 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ list_omap_keys cluster2 mirror rbd_mirroring 2024-09-17T23:40:52.273 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster2 2024-09-17T23:40:52.273 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:40:52.273 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local obj_name=rbd_mirroring 2024-09-17T23:40:52.273 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rados --cluster cluster2 -p mirror listomapkeys rbd_mirroring 2024-09-17T23:40:52.273 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ grep -c image_ 2024-09-17T23:40:52.308 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 0 = 0 2024-09-17T23:40:52.308 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ error_code=0 2024-09-17T23:40:52.308 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set -e 2024-09-17T23:40:52.308 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' 0 -eq 0 ']' 2024-09-17T23:40:52.309 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:40:52.309 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirror_leader image_map 2024-09-17T23:40:52.309 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:40:52.309 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local pool=mirror 2024-09-17T23:40:52.309 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local obj_name=rbd_mirror_leader 2024-09-17T23:40:52.309 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local filter=image_map 2024-09-17T23:40:52.309 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-17T23:40:52.309 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ sleep 0 2024-09-17T23:40:52.310 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set +e 2024-09-17T23:40:52.310 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ count_omap_keys_with_filter cluster2 mirror rbd_mirror_leader image_map 2024-09-17T23:40:52.310 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster2 2024-09-17T23:40:52.310 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:40:52.310 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local obj_name=rbd_mirror_leader 2024-09-17T23:40:52.310 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local filter=image_map 2024-09-17T23:40:52.311 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ list_omap_keys cluster2 mirror rbd_mirror_leader 2024-09-17T23:40:52.311 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local cluster=cluster2 2024-09-17T23:40:52.311 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local pool=mirror 2024-09-17T23:40:52.311 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ local obj_name=rbd_mirror_leader 2024-09-17T23:40:52.311 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ rados --cluster cluster2 -p mirror listomapkeys rbd_mirror_leader 2024-09-17T23:40:52.311 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ grep -c image_map 2024-09-17T23:40:52.345 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test 0 = 0 2024-09-17T23:40:52.345 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ error_code=0 2024-09-17T23:40:52.345 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set -e 2024-09-17T23:40:52.345 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' 0 -eq 0 ']' 2024-09-17T23:40:52.346 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 0 2024-09-17T23:40:52.346 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ cleanup 0 2024-09-17T23:40:52.346 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local error_code=0 2024-09-17T23:40:52.346 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ set +e 2024-09-17T23:40:52.346 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' 0 -ne 0 ']' 2024-09-17T23:40:52.346 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -z '' ']' 2024-09-17T23:40:52.346 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster instance 2024-09-17T23:40:52.346 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ CEPH_ARGS= 2024-09-17T23:40:52.346 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ ceph --cluster cluster1 osd pool rm mirror mirror --yes-i-really-really-mean-it 2024-09-17T23:40:53.460 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi038.stderr:2024-09-17T23:40:53.458+0000 7f9b5c9ba640 -1 rbd::mirror::InstanceWatcher: 0x55bd3b7b4380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5965 object: (2) No such file or directory 2024-09-17T23:40:53.460 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi038.stderr:2024-09-17T23:40:53.458+0000 7f9b5c9ba640 -1 rbd::mirror::InstanceWatcher: 0x55bd3bcae000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5965 object: (2) No such file or directory 2024-09-17T23:40:53.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi038.stderr:2024-09-17T23:40:53.458+0000 7f9b569ae640 -1 rbd::mirror::InstanceWatcher: 0x55bd3b7b4380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-17T23:40:53.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi038.stderr:2024-09-17T23:40:53.458+0000 7f9b569ae640 -1 rbd::mirror::InstanceWatcher: 0x55bd3bcae000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-17T23:40:53.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi038.stderr:2024-09-17T23:40:53.462+0000 7f9b561ad640 -1 rbd::mirror::MirrorStatusUpdater 0x55bd3b828240 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-17T23:40:53.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi038.stderr:2024-09-17T23:40:53.462+0000 7f9b5d1bb640 -1 rbd::mirror::NamespaceReplayer: 0x55bd3b832000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-17T23:40:53.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi038.stderr:2024-09-17T23:40:53.462+0000 7f9b569ae640 -1 rbd::mirror::MirrorStatusUpdater 0x55bd3b828360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-17T23:40:53.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi038.stderr:2024-09-17T23:40:53.462+0000 7f9b5d1bb640 -1 rbd::mirror::NamespaceReplayer: 0x55bd3b8321a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-17T23:40:53.531 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:pool 'mirror' does not exist 2024-09-17T23:40:53.542 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ CEPH_ARGS= 2024-09-17T23:40:53.543 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ ceph --cluster cluster2 osd pool rm mirror mirror --yes-i-really-really-mean-it 2024-09-17T23:40:54.553 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:pool 'mirror' does not exist 2024-09-17T23:40:54.571 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ CEPH_ARGS= 2024-09-17T23:40:54.571 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ ceph --cluster cluster1 osd pool rm mirror_parent mirror_parent --yes-i-really-really-mean-it 2024-09-17T23:40:55.547 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:pool 'mirror_parent' does not exist 2024-09-17T23:40:55.559 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ CEPH_ARGS= 2024-09-17T23:40:55.559 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ ceph --cluster cluster2 osd pool rm mirror_parent mirror_parent --yes-i-really-really-mean-it 2024-09-17T23:40:56.162 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi038.stderr:2024-09-17T23:40:56.160+0000 7f6f5ee50640 -1 rbd::mirror::RemotePoolPoller: 0x55dc1c8bc9c0 handle_mirror_peer_ping: failed to ping remote mirror peer: (2) No such file or directory 2024-09-17T23:40:56.162 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi038.stderr:2024-09-17T23:40:56.160+0000 7f6f5e64f640 -1 rbd::mirror::RemotePoolPoller: 0x55dc1c8bc9c0 handle_mirror_peer_list: failed to retrieve mirror peers: (2) No such file or directory 2024-09-17T23:40:56.162 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi038.stderr:2024-09-17T23:40:56.160+0000 7f6f5e64f640 -1 rbd::mirror::PoolReplayer: 0x55dc1c8cfb00 handle_remote_pool_meta_updated: remote pool metadata updated unexpectedly 2024-09-17T23:40:56.163 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi038.stderr:2024-09-17T23:40:56.160+0000 7f6f6b669640 -1 rbd::mirror::InstanceWatcher: 0x55dc1da54380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5982 object: (2) No such file or directory 2024-09-17T23:40:56.163 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi038.stderr:2024-09-17T23:40:56.160+0000 7f6f6b669640 -1 rbd::mirror::InstanceWatcher: 0x55dc1da54a80 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5982 object: (2) No such file or directory 2024-09-17T23:40:56.163 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi038.stderr:2024-09-17T23:40:56.161+0000 7f6f6565d640 -1 rbd::mirror::InstanceWatcher: 0x55dc1da54380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-17T23:40:56.163 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi038.stderr:2024-09-17T23:40:56.161+0000 7f6f6565d640 -1 rbd::mirror::InstanceWatcher: 0x55dc1da54a80 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-17T23:40:56.164 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi038.stderr:2024-09-17T23:40:56.161+0000 7f6f5ee50640 -1 rbd::mirror::MirrorStatusUpdater 0x55dc1daa25a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-17T23:40:56.164 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi038.stderr:2024-09-17T23:40:56.161+0000 7f6f5ee50640 -1 rbd::mirror::MirrorStatusUpdater 0x55dc1daa2360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-17T23:40:56.164 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi038.stderr:2024-09-17T23:40:56.161+0000 7f6f6b669640 -1 rbd::mirror::NamespaceReplayer: 0x55dc1daa0000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-17T23:40:56.164 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi038.stderr:2024-09-17T23:40:56.161+0000 7f6f6b669640 -1 rbd::mirror::NamespaceReplayer: 0x55dc1daa01a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-17T23:40:56.164 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi038.stderr:2024-09-17T23:40:56.161+0000 7f6f64e5c640 -1 rbd::mirror::MirrorStatusUpdater 0x55dc1daa2000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-17T23:40:56.164 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi038.stderr:2024-09-17T23:40:56.161+0000 7f6f64e5c640 -1 rbd::mirror::MirrorStatusUpdater 0x55dc1daa2120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-17T23:40:56.164 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi038.stderr:2024-09-17T23:40:56.161+0000 7f6f6be6a640 -1 rbd::mirror::NamespaceReplayer: 0x55dc1daa0000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-17T23:40:56.164 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi038.stderr:2024-09-17T23:40:56.161+0000 7f6f6b669640 -1 rbd::mirror::NamespaceReplayer: 0x55dc1daa01a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-17T23:40:56.218 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi038.stderr:2024-09-17T23:40:56.216+0000 7f6f50e34640 -1 rbd::mirror::RemotePoolPoller: 0x55dc1c8bc680 handle_mirror_peer_ping: failed to ping remote mirror peer: (2) No such file or directory 2024-09-17T23:40:56.218 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi038.stderr:2024-09-17T23:40:56.216+0000 7f6f51635640 -1 rbd::mirror::RemotePoolPoller: 0x55dc1c8bc680 handle_mirror_peer_list: failed to retrieve mirror peers: (2) No such file or directory 2024-09-17T23:40:56.218 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi038.stderr:2024-09-17T23:40:56.216+0000 7f6f51635640 -1 rbd::mirror::PoolReplayer: 0x55dc1d9bcf00 handle_remote_pool_meta_updated: remote pool metadata updated unexpectedly 2024-09-17T23:40:56.361 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi038.stderr:2024-09-17T23:40:56.359+0000 7f6f6be6a640 -1 rbd::mirror::LeaderWatcher: 0x55dc1d9bcc00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-17T23:40:56.361 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi038.stderr:2024-09-17T23:40:56.359+0000 7f6f6be6a640 -1 rbd::mirror::InstanceWatcher: 0x55dc1da54000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5982 object: (2) No such file or directory 2024-09-17T23:40:56.362 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi038.stderr:2024-09-17T23:40:56.359+0000 7f6f6565d640 -1 rbd::mirror::InstanceWatcher: 0x55dc1da54000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-17T23:40:56.362 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi038.stderr:2024-09-17T23:40:56.360+0000 7f6f5ee50640 -1 rbd::mirror::MirrorStatusUpdater 0x55dc1d81e5a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-17T23:40:56.362 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi038.stderr:2024-09-17T23:40:56.360+0000 7f6f6be6a640 -1 rbd::mirror::NamespaceReplayer: 0x55dc1d6a61a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-17T23:40:56.362 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi038.stderr:2024-09-17T23:40:56.360+0000 7f6f64e5c640 -1 rbd::mirror::MirrorStatusUpdater 0x55dc1d7c90e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-17T23:40:56.362 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi038.stderr:2024-09-17T23:40:56.360+0000 7f6f6b669640 -1 rbd::mirror::NamespaceReplayer: 0x55dc1d6a61a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-17T23:40:56.364 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi038.stderr:2024-09-17T23:40:56.362+0000 7f6f6f6c8640 -1 rbd::mirror::Mirror: 0x55dc1cb5be60 update_pool_replayers: restarting failed pool replayer for uuid: 7fcde847-128c-4311-ba71-be3574fa77a0 cluster: cluster1 client: client.mirror 2024-09-17T23:40:56.371 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi038.stderr:2024-09-17T23:40:56.369+0000 7f6f6b669640 -1 rbd::mirror::InstanceWatcher: 0x55dc1da54700 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5984 object: (2) No such file or directory 2024-09-17T23:40:56.377 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi038.stderr:2024-09-17T23:40:56.375+0000 7f6f57e42640 -1 rbd::mirror::InstanceWatcher: 0x55dc1da54700 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-17T23:40:56.377 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi038.stderr:2024-09-17T23:40:56.375+0000 7f6f50e34640 -1 rbd::mirror::MirrorStatusUpdater 0x55dc1dc94c60 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-17T23:40:56.377 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi038.stderr:2024-09-17T23:40:56.375+0000 7f6f6be6a640 -1 rbd::mirror::NamespaceReplayer: 0x55dc1df1a000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-17T23:40:56.378 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi038.stderr:2024-09-17T23:40:56.375+0000 7f6f57641640 -1 rbd::mirror::MirrorStatusUpdater 0x55dc1dc96c60 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-17T23:40:56.378 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi038.stderr:2024-09-17T23:40:56.375+0000 7f6f6b669640 -1 rbd::mirror::NamespaceReplayer: 0x55dc1df1a000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-17T23:40:56.408 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi038.stderr:2024-09-17T23:40:56.406+0000 7f6f6f6c8640 -1 rbd::mirror::PoolReplayer: 0x55dc1d9bcf00 init: error accessing local pool 4: (2) No such file or directory 2024-09-17T23:40:56.483 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:pool 'mirror_parent' does not exist 2024-09-17T23:40:56.495 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for cluster in "${CLUSTER1}" "${CLUSTER2}" 2024-09-17T23:40:56.495 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ stop_mirrors cluster1 2024-09-17T23:40:56.495 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1 2024-09-17T23:40:56.495 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local sig= 2024-09-17T23:40:56.495 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ seq 0 3 2024-09-17T23:40:56.499 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:40:56.500 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ stop_mirror cluster1:0 '' 2024-09-17T23:40:56.500 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1:0 2024-09-17T23:40:56.500 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local sig= 2024-09-17T23:40:56.500 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n 1 2024-09-17T23:40:56.500 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 2024-09-17T23:40:56.500 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:40:56.500 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ stop_mirror cluster1:1 '' 2024-09-17T23:40:56.500 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1:1 2024-09-17T23:40:56.500 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local sig= 2024-09-17T23:40:56.500 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n 1 2024-09-17T23:40:56.500 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 2024-09-17T23:40:56.500 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:40:56.500 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ stop_mirror cluster1:2 '' 2024-09-17T23:40:56.500 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1:2 2024-09-17T23:40:56.501 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local sig= 2024-09-17T23:40:56.501 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n 1 2024-09-17T23:40:56.501 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 2024-09-17T23:40:56.501 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:40:56.501 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ stop_mirror cluster1:3 '' 2024-09-17T23:40:56.501 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster1:3 2024-09-17T23:40:56.501 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local sig= 2024-09-17T23:40:56.501 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n 1 2024-09-17T23:40:56.501 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 2024-09-17T23:40:56.501 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for cluster in "${CLUSTER1}" "${CLUSTER2}" 2024-09-17T23:40:56.501 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ stop_mirrors cluster2 2024-09-17T23:40:56.501 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2 2024-09-17T23:40:56.501 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local sig= 2024-09-17T23:40:56.501 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:++ seq 0 3 2024-09-17T23:40:56.501 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:40:56.502 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ stop_mirror cluster2:0 '' 2024-09-17T23:40:56.502 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2:0 2024-09-17T23:40:56.502 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local sig= 2024-09-17T23:40:56.502 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n 1 2024-09-17T23:40:56.502 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 2024-09-17T23:40:56.502 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:40:56.502 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ stop_mirror cluster2:1 '' 2024-09-17T23:40:56.502 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2:1 2024-09-17T23:40:56.502 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local sig= 2024-09-17T23:40:56.502 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n 1 2024-09-17T23:40:56.502 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 2024-09-17T23:40:56.502 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:40:56.502 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ stop_mirror cluster2:2 '' 2024-09-17T23:40:56.502 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2:2 2024-09-17T23:40:56.502 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local sig= 2024-09-17T23:40:56.502 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n 1 2024-09-17T23:40:56.503 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 2024-09-17T23:40:56.503 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:40:56.503 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ stop_mirror cluster2:3 '' 2024-09-17T23:40:56.503 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local cluster=cluster2:3 2024-09-17T23:40:56.503 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ local sig= 2024-09-17T23:40:56.503 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test -n 1 2024-09-17T23:40:56.503 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ return 2024-09-17T23:40:56.503 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' -z 1 ']' 2024-09-17T23:40:56.503 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ test '' = /tmp/tmp.s5MEFDgsej 2024-09-17T23:40:56.503 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ rm -Rf /tmp/tmp.s5MEFDgsej 2024-09-17T23:40:56.504 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stdout:OK 2024-09-17T23:40:56.504 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ '[' 0 -eq 0 ']' 2024-09-17T23:40:56.504 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ echo OK 2024-09-17T23:40:56.505 INFO:tasks.workunit.cluster1.client.mirror.smithi038.stderr:+ exit 0 2024-09-17T23:40:56.505 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-17T23:40:56.505 DEBUG:teuthology.orchestra.run.smithi038:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp 2024-09-17T23:40:56.582 INFO:tasks.workunit:Stopping ['rbd/rbd_mirror_stress.sh'] on cluster1.client.mirror... 2024-09-17T23:40:56.582 DEBUG:teuthology.orchestra.run.smithi038:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.cluster1.client.mirror /home/ubuntu/cephtest/clone.cluster1.client.mirror 2024-09-17T23:40:56.888 DEBUG:teuthology.parallel:result is None 2024-09-17T23:40:56.889 DEBUG:teuthology.orchestra.run.smithi038:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2024-09-17T23:40:56.915 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2024-09-17T23:40:56.915 DEBUG:teuthology.orchestra.run.smithi038:> rmdir -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-09-17T23:40:56.979 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2024-09-17T23:40:56.979 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror-thrash 2024-09-17T23:40:57.125 INFO:tasks.rbd_mirror_thrash:joining rbd_mirror_thrash 2024-09-17T23:40:57.126 INFO:tasks.rbd_mirror_thrash:stat['kill'] = 18 2024-09-17T23:40:57.126 INFO:tasks.rbd_mirror_thrash:done joining 2024-09-17T23:40:57.126 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror-thrash 2024-09-17T23:40:57.144 INFO:tasks.rbd_mirror_thrash:joining rbd_mirror_thrash 2024-09-17T23:40:57.144 INFO:tasks.rbd_mirror_thrash:stat['kill'] = 15 2024-09-17T23:40:57.145 INFO:tasks.rbd_mirror_thrash:done joining 2024-09-17T23:40:57.145 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-09-17T23:40:57.157 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.3:waiting for process to exit 2024-09-17T23:40:57.157 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T23:40:57.158 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:40:57.156+0000 7fcc9acf2640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 69520) UID: 1000 2024-09-17T23:40:57.159 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:40:57.157+0000 7fcc85cc8640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x55d1644fc480 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-09-17T23:40:57.159 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:40:57.157+0000 7fcc864c9640 -1 rbd::mirror::PoolWatcher: 0x55d16284fc20 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-17T23:40:57.159 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:40:57.157+0000 7fcc9a4f1640 -1 rbd::mirror::InstanceWatcher: 0x55d163798380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6642 object: (2) No such file or directory 2024-09-17T23:40:57.159 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:40:57.157+0000 7fcc9a4f1640 -1 rbd::mirror::InstanceWatcher: 0x55d163c5c380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6642 object: (2) No such file or directory 2024-09-17T23:40:57.161 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:40:57.157+0000 7fcc93ce4640 -1 rbd::mirror::InstanceWatcher: 0x55d163798380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-17T23:40:57.161 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:40:57.157+0000 7fcc93ce4640 -1 rbd::mirror::InstanceWatcher: 0x55d163c5c380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-17T23:40:57.161 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:40:57.157+0000 7fcc99cf0640 -1 rbd::mirror::LeaderWatcher: 0x55d163975200 handle_notify_lock_released: error notifying leader lock released: (2) No such file or directory 2024-09-17T23:40:57.161 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:40:57.157+0000 7fcc99cf0640 -1 rbd::mirror::LeaderWatcher: 0x55d163975200 handle_shut_down_leader_lock: error shutting down leader lock: (2) No such file or directory 2024-09-17T23:40:57.161 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:40:57.158+0000 7fcc9a4f1640 -1 rbd::mirror::LeaderWatcher: 0x55d163975200 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-17T23:40:57.161 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:40:57.158+0000 7fcc8ccd6640 -1 rbd::mirror::MirrorStatusUpdater 0x55d1638025a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-17T23:40:57.161 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:40:57.158+0000 7fcc9a4f1640 -1 rbd::mirror::NamespaceReplayer: 0x55d163790340 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-17T23:40:57.161 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:40:57.158+0000 7fcc8d4d7640 -1 rbd::mirror::MirrorStatusUpdater 0x55d163802360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-17T23:40:57.161 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:40:57.158+0000 7fcc9a4f1640 -1 rbd::mirror::InstanceWatcher: 0x55d163c5c000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6645 object: (2) No such file or directory 2024-09-17T23:40:57.161 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:40:57.158+0000 7fcc9a4f1640 -1 rbd::mirror::NamespaceReplayer: 0x55d1637904e0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-17T23:40:57.161 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:40:57.158+0000 7fcc934e3640 -1 rbd::mirror::MirrorStatusUpdater 0x55d163802000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-17T23:40:57.161 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:40:57.158+0000 7fcc99cf0640 -1 rbd::mirror::NamespaceReplayer: 0x55d163790340 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-17T23:40:57.161 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:40:57.158+0000 7fcc93ce4640 -1 rbd::mirror::MirrorStatusUpdater 0x55d163802120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-17T23:40:57.161 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:40:57.158+0000 7fcc99cf0640 -1 rbd::mirror::NamespaceReplayer: 0x55d1637904e0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-17T23:40:57.162 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:40:57.158+0000 7fcc85cc8640 -1 rbd::mirror::InstanceWatcher: 0x55d163c5c000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-17T23:40:57.162 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:40:57.160+0000 7fcc864c9640 -1 rbd::mirror::MirrorStatusUpdater 0x55d163a0b320 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-17T23:40:57.162 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:40:57.160+0000 7fcc99cf0640 -1 rbd::mirror::NamespaceReplayer: 0x55d163790820 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-17T23:40:57.164 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:40:57.162+0000 7fcc9a4f1640 -1 rbd::mirror::LeaderWatcher: 0x55d1645c6c00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-17T23:40:57.164 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:40:57.162+0000 7fcc9a4f1640 -1 rbd::mirror::InstanceWatcher: 0x55d163798000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6642 object: (2) No such file or directory 2024-09-17T23:40:57.164 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:40:57.163+0000 7fcc934e3640 -1 rbd::mirror::InstanceWatcher: 0x55d163798000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-17T23:40:57.165 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:40:57.163+0000 7fcc8ccd6640 -1 rbd::mirror::MirrorStatusUpdater 0x55d1643be7e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-17T23:40:57.165 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:40:57.163+0000 7fcc99cf0640 -1 rbd::mirror::NamespaceReplayer: 0x55d163790000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-17T23:40:57.165 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:40:57.163+0000 7fcc93ce4640 -1 rbd::mirror::MirrorStatusUpdater 0x55d164378fc0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-17T23:40:57.165 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi038.stderr:2024-09-17T23:40:57.163+0000 7fcc9a4f1640 -1 rbd::mirror::NamespaceReplayer: 0x55d163790000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-17T23:40:57.176 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Stopped 2024-09-17T23:40:57.177 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-09-17T23:40:57.186 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.2:waiting for process to exit 2024-09-17T23:40:57.186 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T23:40:57.218 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi038.stderr:2024-09-17T23:40:57.216+0000 7f6f6c66b640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 61078) UID: 1000 2024-09-17T23:40:57.229 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Stopped 2024-09-17T23:40:57.229 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-09-17T23:40:57.239 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.1:waiting for process to exit 2024-09-17T23:40:57.239 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T23:40:57.252 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi038.stderr:2024-09-17T23:40:57.250+0000 7efee6be8640 -1 rbd::mirror::LeaderWatcher: 0x5562629e2c00 handle_notify_heartbeat: error notifying heartbeat: (2) No such file or directory, releasing leader 2024-09-17T23:40:57.252 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi038.stderr:2024-09-17T23:40:57.250+0000 7efee03db640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x556262ea2d80 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-09-17T23:40:57.252 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi038.stderr:2024-09-17T23:40:57.250+0000 7efee03db640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x556262ea2480 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-09-17T23:40:57.252 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi038.stderr:2024-09-17T23:40:57.250+0000 7efee03db640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x5562629ec480 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-09-17T23:40:57.253 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi038.stderr:2024-09-17T23:40:57.250+0000 7f65190f0640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 71202) UID: 1000 2024-09-17T23:40:57.253 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi038.stderr:2024-09-17T23:40:57.250+0000 7efedfbda640 -1 rbd::mirror::PoolWatcher: 0x5562631301e0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-17T23:40:57.253 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi038.stderr:2024-09-17T23:40:57.250+0000 7efedfbda640 -1 rbd::mirror::PoolWatcher: 0x55626311c3c0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-17T23:40:57.253 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi038.stderr:2024-09-17T23:40:57.250+0000 7efee03db640 -1 rbd::mirror::PoolWatcher: 0x55626303b2c0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-17T23:40:57.253 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi038.stderr:2024-09-17T23:40:57.250+0000 7efed93cd640 -1 rbd::mirror::PoolWatcher: 0x556263130b40 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-17T23:40:57.253 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi038.stderr:2024-09-17T23:40:57.250+0000 7efed93cd640 -1 rbd::mirror::PoolWatcher: 0x556263130960 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-17T23:40:57.254 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi038.stderr:2024-09-17T23:40:57.250+0000 7efed93cd640 -1 rbd::mirror::PoolWatcher: 0x5562631303c0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-17T23:40:57.254 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi038.stderr:2024-09-17T23:40:57.251+0000 7f65180ee640 -1 rbd::mirror::InstanceWatcher: 0x55f0a1d82380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6759 object: (2) No such file or directory 2024-09-17T23:40:57.254 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi038.stderr:2024-09-17T23:40:57.251+0000 7f65180ee640 -1 rbd::mirror::InstanceWatcher: 0x55f0a1d82a80 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6759 object: (2) No such file or directory 2024-09-17T23:40:57.255 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi038.stderr:2024-09-17T23:40:57.251+0000 7efee63e7640 -1 rbd::mirror::LeaderWatcher: 0x5562629e2c00 handle_notify_lock_released: error notifying leader lock released: (2) No such file or directory 2024-09-17T23:40:57.255 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi038.stderr:2024-09-17T23:40:57.251+0000 7efee63e7640 -1 rbd::mirror::LeaderWatcher: 0x5562629e2c00 handle_release_leader_lock: error releasing lock: (2) No such file or directory 2024-09-17T23:40:57.256 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi038.stderr:2024-09-17T23:40:57.251+0000 7f65118e1640 -1 rbd::mirror::InstanceWatcher: 0x55f0a1d82380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-17T23:40:57.256 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi038.stderr:2024-09-17T23:40:57.251+0000 7f65118e1640 -1 rbd::mirror::InstanceWatcher: 0x55f0a1d82a80 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-17T23:40:57.256 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi038.stderr:2024-09-17T23:40:57.252+0000 7f65120e2640 -1 rbd::mirror::MirrorStatusUpdater 0x55f0a1dc8000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-17T23:40:57.256 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi038.stderr:2024-09-17T23:40:57.252+0000 7f65118e1640 -1 rbd::mirror::MirrorStatusUpdater 0x55f0a1dc8120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-17T23:40:57.256 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi038.stderr:2024-09-17T23:40:57.252+0000 7f65180ee640 -1 rbd::mirror::NamespaceReplayer: 0x55f0a1dc4000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-17T23:40:57.256 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi038.stderr:2024-09-17T23:40:57.252+0000 7f65188ef640 -1 rbd::mirror::NamespaceReplayer: 0x55f0a1dc41a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-17T23:40:57.256 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi038.stderr:2024-09-17T23:40:57.252+0000 7f65180ee640 -1 rbd::mirror::InstanceWatcher: 0x55f0a225e000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6762 object: (2) No such file or directory 2024-09-17T23:40:57.256 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi038.stderr:2024-09-17T23:40:57.253+0000 7f65048c7640 -1 rbd::mirror::InstanceWatcher: 0x55f0a225e000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-17T23:40:57.256 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi038.stderr:2024-09-17T23:40:57.253+0000 7f64fe0ba640 -1 rbd::mirror::MirrorStatusUpdater 0x55f0a20086c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-17T23:40:57.256 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi038.stderr:2024-09-17T23:40:57.253+0000 7f65180ee640 -1 rbd::mirror::NamespaceReplayer: 0x55f0a2252000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-17T23:40:57.256 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi038.stderr:2024-09-17T23:40:57.253+0000 7f65040c6640 -1 rbd::mirror::MirrorStatusUpdater 0x55f0a1fbd200 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-17T23:40:57.256 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi038.stderr:2024-09-17T23:40:57.253+0000 7f65188ef640 -1 rbd::mirror::NamespaceReplayer: 0x55f0a2252000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-17T23:40:57.258 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi038.stderr:2024-09-17T23:40:57.255+0000 7f65188ef640 -1 rbd::mirror::LeaderWatcher: 0x55f0a1ce8c00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-17T23:40:57.258 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi038.stderr:2024-09-17T23:40:57.256+0000 7f65180ee640 -1 rbd::mirror::InstanceWatcher: 0x55f0a1d82000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6759 object: (2) No such file or directory 2024-09-17T23:40:57.258 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi038.stderr:2024-09-17T23:40:57.256+0000 7f65120e2640 -1 rbd::mirror::InstanceWatcher: 0x55f0a1d82000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-17T23:40:57.258 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi038.stderr:2024-09-17T23:40:57.256+0000 7f65118e1640 -1 rbd::mirror::MirrorStatusUpdater 0x55f0a1b35200 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-17T23:40:57.258 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi038.stderr:2024-09-17T23:40:57.256+0000 7f65188ef640 -1 rbd::mirror::NamespaceReplayer: 0x55f0a19e61a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-17T23:40:57.269 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Stopped 2024-09-17T23:40:57.269 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-09-17T23:40:57.279 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.0:waiting for process to exit 2024-09-17T23:40:57.279 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T23:40:57.310 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi038.stderr:2024-09-17T23:40:57.308+0000 7fc2d62d9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 70302) UID: 1000 2024-09-17T23:40:57.311 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi038.stderr:2024-09-17T23:40:57.308+0000 7fc2ceaca640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x55b7a5ca8d80 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-09-17T23:40:57.311 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi038.stderr:2024-09-17T23:40:57.308+0000 7fc2ceaca640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x55b7a5ca8480 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-09-17T23:40:57.311 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi038.stderr:2024-09-17T23:40:57.309+0000 7fc2cf2cb640 -1 rbd::mirror::PoolWatcher: 0x55b7a4bb1680 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-17T23:40:57.311 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi038.stderr:2024-09-17T23:40:57.309+0000 7fc2ceaca640 -1 rbd::mirror::PoolWatcher: 0x55b7a4bb1860 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-17T23:40:57.311 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi038.stderr:2024-09-17T23:40:57.309+0000 7fc2c1ab0640 -1 rbd::mirror::InstanceWatcher: 0x55b7a5d50a80 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-17T23:40:57.312 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi038.stderr:2024-09-17T23:40:57.309+0000 7fc2c82bd640 -1 rbd::mirror::PoolWatcher: 0x55b7a62b1a40 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-17T23:40:57.312 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi038.stderr:2024-09-17T23:40:57.309+0000 7fc2c82bd640 -1 rbd::mirror::PoolWatcher: 0x55b7a62b05a0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-17T23:40:57.312 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi038.stderr:2024-09-17T23:40:57.310+0000 7fc2d5ad8640 -1 rbd::mirror::InstanceWatcher: 0x55b7a5d50380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6701 object: (2) No such file or directory 2024-09-17T23:40:57.312 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi038.stderr:2024-09-17T23:40:57.310+0000 7fc2d5ad8640 -1 rbd::mirror::InstanceWatcher: 0x55b7a5d50700 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6701 object: (2) No such file or directory 2024-09-17T23:40:57.313 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi038.stderr:2024-09-17T23:40:57.310+0000 7fc2ceaca640 -1 rbd::mirror::InstanceWatcher: 0x55b7a5d50380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-17T23:40:57.313 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi038.stderr:2024-09-17T23:40:57.310+0000 7fc2cf2cb640 -1 rbd::mirror::InstanceWatcher: 0x55b7a5d50700 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-17T23:40:57.314 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi038.stderr:2024-09-17T23:40:57.310+0000 7fc2c8abe640 -1 rbd::mirror::MirrorStatusUpdater 0x55b7a5da0360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-17T23:40:57.314 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi038.stderr:2024-09-17T23:40:57.310+0000 7fc2c8abe640 -1 rbd::mirror::MirrorStatusUpdater 0x55b7a5da05a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-17T23:40:57.314 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi038.stderr:2024-09-17T23:40:57.310+0000 7fc2d52d7640 -1 rbd::mirror::NamespaceReplayer: 0x55b7a5d9e000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-17T23:40:57.314 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi038.stderr:2024-09-17T23:40:57.310+0000 7fc2d52d7640 -1 rbd::mirror::NamespaceReplayer: 0x55b7a5d9e1a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-17T23:40:57.314 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi038.stderr:2024-09-17T23:40:57.311+0000 7fc2ceaca640 -1 rbd::mirror::MirrorStatusUpdater 0x55b7a5da0000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-17T23:40:57.315 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi038.stderr:2024-09-17T23:40:57.311+0000 7fc2ceaca640 -1 rbd::mirror::MirrorStatusUpdater 0x55b7a5da0120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-17T23:40:57.315 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi038.stderr:2024-09-17T23:40:57.311+0000 7fc2d5ad8640 -1 rbd::mirror::NamespaceReplayer: 0x55b7a5d9e000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-17T23:40:57.315 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi038.stderr:2024-09-17T23:40:57.311+0000 7fc2d5ad8640 -1 rbd::mirror::NamespaceReplayer: 0x55b7a5d9e1a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-17T23:40:57.315 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi038.stderr:2024-09-17T23:40:57.311+0000 7fc2cf2cb640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x55b7a617d680 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-09-17T23:40:57.315 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi038.stderr:2024-09-17T23:40:57.312+0000 7fc2ceaca640 -1 rbd::mirror::PoolWatcher: 0x55b7a623c960 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-17T23:40:57.315 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi038.stderr:2024-09-17T23:40:57.312+0000 7fc2c82bd640 -1 rbd::mirror::PoolWatcher: 0x55b7a623de00 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-17T23:40:57.315 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi038.stderr:2024-09-17T23:40:57.312+0000 7fc2d5ad8640 -1 rbd::mirror::LeaderWatcher: 0x55b7a5cc4c00 handle_notify_lock_released: error notifying leader lock released: (2) No such file or directory 2024-09-17T23:40:57.315 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi038.stderr:2024-09-17T23:40:57.312+0000 7fc2d5ad8640 -1 rbd::mirror::LeaderWatcher: 0x55b7a5cc4c00 handle_shut_down_leader_lock: error shutting down leader lock: (2) No such file or directory 2024-09-17T23:40:57.315 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi038.stderr:2024-09-17T23:40:57.312+0000 7fc2d52d7640 -1 rbd::mirror::LeaderWatcher: 0x55b7a5cc4c00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-17T23:40:57.316 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi038.stderr:2024-09-17T23:40:57.313+0000 7fc2d52d7640 -1 rbd::mirror::InstanceWatcher: 0x55b7a5d50000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6701 object: (2) No such file or directory 2024-09-17T23:40:57.316 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi038.stderr:2024-09-17T23:40:57.313+0000 7fc2ceaca640 -1 rbd::mirror::InstanceWatcher: 0x55b7a5d50000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-17T23:40:57.316 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi038.stderr:2024-09-17T23:40:57.313+0000 7fc2c8abe640 -1 rbd::mirror::MirrorStatusUpdater 0x55b7a5b006c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-17T23:40:57.316 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi038.stderr:2024-09-17T23:40:57.313+0000 7fc2d52d7640 -1 rbd::mirror::NamespaceReplayer: 0x55b7a59a61a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-17T23:40:57.316 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi038.stderr:2024-09-17T23:40:57.313+0000 7fc2cf2cb640 -1 rbd::mirror::MirrorStatusUpdater 0x55b7a5ab4fc0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-17T23:40:57.316 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi038.stderr:2024-09-17T23:40:57.313+0000 7fc2d5ad8640 -1 rbd::mirror::NamespaceReplayer: 0x55b7a59a61a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-17T23:40:57.327 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Stopped 2024-09-17T23:40:57.327 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-09-17T23:40:57.337 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.3:waiting for process to exit 2024-09-17T23:40:57.337 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T23:40:57.368 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi038.stderr:2024-09-17T23:40:57.366+0000 7f20b5452640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 47801) UID: 1000 2024-09-17T23:40:57.369 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi038.stderr:2024-09-17T23:40:57.367+0000 7f20b4c51640 -1 rbd::mirror::LeaderWatcher: 0x55a0a39e1200 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-17T23:40:57.369 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi038.stderr:2024-09-17T23:40:57.367+0000 7f20b4450640 -1 rbd::mirror::InstanceWatcher: 0x55a0a37de380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.4764 object: (2) No such file or directory 2024-09-17T23:40:57.369 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi038.stderr:2024-09-17T23:40:57.367+0000 7f20b4450640 -1 rbd::mirror::InstanceWatcher: 0x55a0a37de700 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.4764 object: (2) No such file or directory 2024-09-17T23:40:57.369 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi038.stderr:2024-09-17T23:40:57.367+0000 7f20b4450640 -1 rbd::mirror::InstanceWatcher: 0x55a0a3cb2000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.4767 object: (2) No such file or directory 2024-09-17T23:40:57.369 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi038.stderr:2024-09-17T23:40:57.367+0000 7f20ae444640 -1 rbd::mirror::InstanceWatcher: 0x55a0a37de700 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-17T23:40:57.369 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi038.stderr:2024-09-17T23:40:57.367+0000 7f20ae444640 -1 rbd::mirror::InstanceWatcher: 0x55a0a37de380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-17T23:40:57.370 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi038.stderr:2024-09-17T23:40:57.367+0000 7f20a0428640 -1 rbd::mirror::InstanceWatcher: 0x55a0a3cb2000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-17T23:40:57.371 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi038.stderr:2024-09-17T23:40:57.368+0000 7f20a7c37640 -1 rbd::mirror::MirrorStatusUpdater 0x55a0a3828480 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-17T23:40:57.371 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi038.stderr:2024-09-17T23:40:57.368+0000 7f20a7c37640 -1 rbd::mirror::MirrorStatusUpdater 0x55a0a3828360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-17T23:40:57.371 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi038.stderr:2024-09-17T23:40:57.368+0000 7f20b4c51640 -1 rbd::mirror::NamespaceReplayer: 0x55a0a38261a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-17T23:40:57.371 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi038.stderr:2024-09-17T23:40:57.368+0000 7f20b4c51640 -1 rbd::mirror::NamespaceReplayer: 0x55a0a3826000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-17T23:40:57.371 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi038.stderr:2024-09-17T23:40:57.368+0000 7f20ae444640 -1 rbd::mirror::MirrorStatusUpdater 0x55a0a3828120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-17T23:40:57.371 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi038.stderr:2024-09-17T23:40:57.368+0000 7f20ae444640 -1 rbd::mirror::MirrorStatusUpdater 0x55a0a3828000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-17T23:40:57.371 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi038.stderr:2024-09-17T23:40:57.368+0000 7f20b4450640 -1 rbd::mirror::NamespaceReplayer: 0x55a0a38261a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-17T23:40:57.371 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi038.stderr:2024-09-17T23:40:57.368+0000 7f20b4450640 -1 rbd::mirror::NamespaceReplayer: 0x55a0a3826000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-17T23:40:57.372 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi038.stderr:2024-09-17T23:40:57.369+0000 7f20a0c29640 -1 rbd::mirror::MirrorStatusUpdater 0x55a0a3a07200 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-17T23:40:57.372 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi038.stderr:2024-09-17T23:40:57.369+0000 7f20b4450640 -1 rbd::mirror::NamespaceReplayer: 0x55a0a3b501a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-17T23:40:57.373 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi038.stderr:2024-09-17T23:40:57.372+0000 7f20b4c51640 -1 rbd::mirror::LeaderWatcher: 0x55a0a3746c00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-17T23:40:57.374 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi038.stderr:2024-09-17T23:40:57.372+0000 7f20b4c51640 -1 rbd::mirror::InstanceWatcher: 0x55a0a37de000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.4764 object: (2) No such file or directory 2024-09-17T23:40:57.374 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi038.stderr:2024-09-17T23:40:57.372+0000 7f20adc43640 -1 rbd::mirror::InstanceWatcher: 0x55a0a37de000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-17T23:40:57.375 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi038.stderr:2024-09-17T23:40:57.373+0000 7f20a7436640 -1 rbd::mirror::MirrorStatusUpdater 0x55a0a3734480 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-17T23:40:57.376 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi038.stderr:2024-09-17T23:40:57.373+0000 7f20b4c51640 -1 rbd::mirror::NamespaceReplayer: 0x55a0a34c2000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-17T23:40:57.376 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi038.stderr:2024-09-17T23:40:57.374+0000 7f20ae444640 -1 rbd::mirror::MirrorStatusUpdater 0x55a0a358b200 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-17T23:40:57.376 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi038.stderr:2024-09-17T23:40:57.374+0000 7f20b4450640 -1 rbd::mirror::NamespaceReplayer: 0x55a0a34c2000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-17T23:40:57.388 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Stopped 2024-09-17T23:40:57.388 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-09-17T23:40:57.398 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.2:waiting for process to exit 2024-09-17T23:40:57.398 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T23:40:57.429 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi038.stderr:2024-09-17T23:40:57.427+0000 7efee73e9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 49731) UID: 1000 2024-09-17T23:40:57.430 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi038.stderr:2024-09-17T23:40:57.428+0000 7efed23bf640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x5562647a66c0 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-09-17T23:40:57.430 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi038.stderr:2024-09-17T23:40:57.428+0000 7efee6be8640 -1 rbd::mirror::InstanceWatcher: 0x556262a7c380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.4789 object: (2) No such file or directory 2024-09-17T23:40:57.430 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi038.stderr:2024-09-17T23:40:57.428+0000 7efee6be8640 -1 rbd::mirror::InstanceWatcher: 0x556262f54000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.4789 object: (2) No such file or directory 2024-09-17T23:40:57.430 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi038.stderr:2024-09-17T23:40:57.428+0000 7efed23bf640 -1 rbd::mirror::PoolWatcher: 0x5562618652c0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-17T23:40:57.430 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi038.stderr:2024-09-17T23:40:57.428+0000 7efedfbda640 -1 rbd::mirror::InstanceWatcher: 0x556262a7c380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-17T23:40:57.430 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi038.stderr:2024-09-17T23:40:57.428+0000 7efedfbda640 -1 rbd::mirror::InstanceWatcher: 0x556262f54000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-17T23:40:57.431 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi038.stderr:2024-09-17T23:40:57.428+0000 7efed9bce640 -1 rbd::mirror::MirrorStatusUpdater 0x556262acc360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-17T23:40:57.431 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi038.stderr:2024-09-17T23:40:57.428+0000 7efed9bce640 -1 rbd::mirror::MirrorStatusUpdater 0x556262acc5a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-17T23:40:57.431 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi038.stderr:2024-09-17T23:40:57.428+0000 7efee6be8640 -1 rbd::mirror::NamespaceReplayer: 0x556262aca000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-17T23:40:57.431 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi038.stderr:2024-09-17T23:40:57.428+0000 7efee6be8640 -1 rbd::mirror::NamespaceReplayer: 0x556262aca1a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-17T23:40:57.431 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi038.stderr:2024-09-17T23:40:57.429+0000 7efee03db640 -1 rbd::mirror::MirrorStatusUpdater 0x556262acc000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-17T23:40:57.431 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi038.stderr:2024-09-17T23:40:57.429+0000 7efee03db640 -1 rbd::mirror::MirrorStatusUpdater 0x556262acc120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-17T23:40:57.431 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi038.stderr:2024-09-17T23:40:57.429+0000 7efee63e7640 -1 rbd::mirror::NamespaceReplayer: 0x556262aca000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-17T23:40:57.432 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi038.stderr:2024-09-17T23:40:57.429+0000 7efee63e7640 -1 rbd::mirror::NamespaceReplayer: 0x556262aca1a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-17T23:40:57.432 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi038.stderr:2024-09-17T23:40:57.430+0000 7efee6be8640 -1 rbd::mirror::LeaderWatcher: 0x556262df8600 handle_notify_lock_released: error notifying leader lock released: (2) No such file or directory 2024-09-17T23:40:57.433 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi038.stderr:2024-09-17T23:40:57.430+0000 7efee6be8640 -1 rbd::mirror::LeaderWatcher: 0x556262df8600 handle_shut_down_leader_lock: error shutting down leader lock: (2) No such file or directory 2024-09-17T23:40:57.433 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi038.stderr:2024-09-17T23:40:57.430+0000 7efee63e7640 -1 rbd::mirror::LeaderWatcher: 0x556262df8600 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-17T23:40:57.433 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi038.stderr:2024-09-17T23:40:57.430+0000 7efee63e7640 -1 rbd::mirror::InstanceWatcher: 0x556262a7dc00 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.4792 object: (2) No such file or directory 2024-09-17T23:40:57.433 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi038.stderr:2024-09-17T23:40:57.431+0000 7efed23bf640 -1 rbd::mirror::InstanceWatcher: 0x556262a7dc00 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-17T23:40:57.434 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi038.stderr:2024-09-17T23:40:57.432+0000 7efed2bc0640 -1 rbd::mirror::MirrorStatusUpdater 0x556262ca7200 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-17T23:40:57.434 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi038.stderr:2024-09-17T23:40:57.432+0000 7efee63e7640 -1 rbd::mirror::NamespaceReplayer: 0x556262f44000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-17T23:40:57.437 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi038.stderr:2024-09-17T23:40:57.435+0000 7efee63e7640 -1 rbd::mirror::LeaderWatcher: 0x5562629e2c00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-17T23:40:57.437 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi038.stderr:2024-09-17T23:40:57.435+0000 7efee63e7640 -1 rbd::mirror::InstanceWatcher: 0x556262a7c000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.4789 object: (2) No such file or directory 2024-09-17T23:40:57.438 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi038.stderr:2024-09-17T23:40:57.435+0000 7efedfbda640 -1 rbd::mirror::InstanceWatcher: 0x556262a7c000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-17T23:40:57.438 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi038.stderr:2024-09-17T23:40:57.436+0000 7efed93cd640 -1 rbd::mirror::MirrorStatusUpdater 0x5562628426c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-17T23:40:57.438 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi038.stderr:2024-09-17T23:40:57.436+0000 7efee6be8640 -1 rbd::mirror::NamespaceReplayer: 0x5562632ba000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-17T23:40:57.438 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi038.stderr:2024-09-17T23:40:57.436+0000 7efee03db640 -1 rbd::mirror::MirrorStatusUpdater 0x5562627a30e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-17T23:40:57.438 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi038.stderr:2024-09-17T23:40:57.436+0000 7efee63e7640 -1 rbd::mirror::NamespaceReplayer: 0x5562632ba000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-17T23:40:57.452 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Stopped 2024-09-17T23:40:57.452 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-09-17T23:40:57.463 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.1:waiting for process to exit 2024-09-17T23:40:57.463 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T23:40:57.493 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi038.stderr:2024-09-17T23:40:57.491+0000 7f1a6c8f2640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 70659) UID: 1000 2024-09-17T23:40:57.493 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi038.stderr:2024-09-17T23:40:57.492+0000 7f1a6b8f0640 -1 rbd::mirror::LeaderWatcher: 0x5620a1675200 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-17T23:40:57.495 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi038.stderr:2024-09-17T23:40:57.492+0000 7f1a6b8f0640 -1 rbd::mirror::InstanceWatcher: 0x5620a145e380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6079 object: (2) No such file or directory 2024-09-17T23:40:57.495 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi038.stderr:2024-09-17T23:40:57.492+0000 7f1a6b8f0640 -1 rbd::mirror::InstanceWatcher: 0x5620a145ea80 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6079 object: (2) No such file or directory 2024-09-17T23:40:57.495 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi038.stderr:2024-09-17T23:40:57.492+0000 7f1a6b8f0640 -1 rbd::mirror::InstanceWatcher: 0x5620a145e700 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6081 object: (2) No such file or directory 2024-09-17T23:40:57.495 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi038.stderr:2024-09-17T23:40:57.492+0000 7f1a658e4640 -1 rbd::mirror::InstanceWatcher: 0x5620a145e380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-17T23:40:57.495 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi038.stderr:2024-09-17T23:40:57.492+0000 7f1a658e4640 -1 rbd::mirror::InstanceWatcher: 0x5620a145ea80 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-17T23:40:57.495 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi038.stderr:2024-09-17T23:40:57.492+0000 7f1a580c9640 -1 rbd::mirror::InstanceWatcher: 0x5620a145e700 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-17T23:40:57.495 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi038.stderr:2024-09-17T23:40:57.493+0000 7f1a5f0d7640 -1 rbd::mirror::MirrorStatusUpdater 0x5620a14c25a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-17T23:40:57.495 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi038.stderr:2024-09-17T23:40:57.493+0000 7f1a5f0d7640 -1 rbd::mirror::MirrorStatusUpdater 0x5620a14c2360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-17T23:40:57.496 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi038.stderr:2024-09-17T23:40:57.493+0000 7f1a6b8f0640 -1 rbd::mirror::NamespaceReplayer: 0x5620a14c0000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-17T23:40:57.496 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi038.stderr:2024-09-17T23:40:57.493+0000 7f1a6b8f0640 -1 rbd::mirror::NamespaceReplayer: 0x5620a14c01a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-17T23:40:57.496 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi038.stderr:2024-09-17T23:40:57.493+0000 7f1a658e4640 -1 rbd::mirror::MirrorStatusUpdater 0x5620a14c2000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-17T23:40:57.496 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi038.stderr:2024-09-17T23:40:57.493+0000 7f1a658e4640 -1 rbd::mirror::MirrorStatusUpdater 0x5620a14c2120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-17T23:40:57.496 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi038.stderr:2024-09-17T23:40:57.493+0000 7f1a6c0f1640 -1 rbd::mirror::NamespaceReplayer: 0x5620a14c0000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-17T23:40:57.496 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi038.stderr:2024-09-17T23:40:57.493+0000 7f1a6c0f1640 -1 rbd::mirror::NamespaceReplayer: 0x5620a14c01a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-17T23:40:57.496 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi038.stderr:2024-09-17T23:40:57.494+0000 7f1a578c8640 -1 rbd::mirror::MirrorStatusUpdater 0x5620a16a7200 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-17T23:40:57.496 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi038.stderr:2024-09-17T23:40:57.494+0000 7f1a6c0f1640 -1 rbd::mirror::NamespaceReplayer: 0x5620a1922000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-17T23:40:57.498 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi038.stderr:2024-09-17T23:40:57.496+0000 7f1a6c0f1640 -1 rbd::mirror::LeaderWatcher: 0x5620a13bec00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-17T23:40:57.499 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi038.stderr:2024-09-17T23:40:57.497+0000 7f1a6c0f1640 -1 rbd::mirror::InstanceWatcher: 0x5620a145e000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6079 object: (2) No such file or directory 2024-09-17T23:40:57.500 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi038.stderr:2024-09-17T23:40:57.497+0000 7f1a650e3640 -1 rbd::mirror::InstanceWatcher: 0x5620a145e000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-17T23:40:57.500 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi038.stderr:2024-09-17T23:40:57.498+0000 7f1a5e8d6640 -1 rbd::mirror::MirrorStatusUpdater 0x5620a12166c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-17T23:40:57.500 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi038.stderr:2024-09-17T23:40:57.498+0000 7f1a6c0f1640 -1 rbd::mirror::NamespaceReplayer: 0x5620a028a680 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-17T23:40:57.500 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi038.stderr:2024-09-17T23:40:57.498+0000 7f1a658e4640 -1 rbd::mirror::MirrorStatusUpdater 0x5620a11b70e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-17T23:40:57.500 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi038.stderr:2024-09-17T23:40:57.498+0000 7f1a6b8f0640 -1 rbd::mirror::NamespaceReplayer: 0x5620a028a680 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-17T23:40:57.511 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Stopped 2024-09-17T23:40:57.511 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-09-17T23:40:57.522 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.0:waiting for process to exit 2024-09-17T23:40:57.522 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T23:40:57.552 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi038.stderr:2024-09-17T23:40:57.550+0000 7f9b5d9bc640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 68436) UID: 1000 2024-09-17T23:40:57.553 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi038.stderr:2024-09-17T23:40:57.551+0000 7f9b5d1bb640 -1 rbd::mirror::LeaderWatcher: 0x55bd3b99cf00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-17T23:40:57.553 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi038.stderr:2024-09-17T23:40:57.551+0000 7f9b5c9ba640 -1 rbd::mirror::InstanceWatcher: 0x55bd3b7b5c00 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5968 object: (2) No such file or directory 2024-09-17T23:40:57.554 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi038.stderr:2024-09-17T23:40:57.552+0000 7f9b49193640 -1 rbd::mirror::InstanceWatcher: 0x55bd3b7b5c00 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-17T23:40:57.555 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi038.stderr:2024-09-17T23:40:57.553+0000 7f9b48992640 -1 rbd::mirror::MirrorStatusUpdater 0x55bd3ba00fc0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-17T23:40:57.556 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi038.stderr:2024-09-17T23:40:57.553+0000 7f9b5c9ba640 -1 rbd::mirror::NamespaceReplayer: 0x55bd3b8324e0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-17T23:40:57.558 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi038.stderr:2024-09-17T23:40:57.556+0000 7f9b5c9ba640 -1 rbd::mirror::LeaderWatcher: 0x55bd3b70f500 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-17T23:40:57.559 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi038.stderr:2024-09-17T23:40:57.556+0000 7f9b5d1bb640 -1 rbd::mirror::InstanceWatcher: 0x55bd3b7b4000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5965 object: (2) No such file or directory 2024-09-17T23:40:57.559 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi038.stderr:2024-09-17T23:40:57.556+0000 7f9b561ad640 -1 rbd::mirror::InstanceWatcher: 0x55bd3b7b4000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-17T23:40:57.559 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi038.stderr:2024-09-17T23:40:57.557+0000 7f9b4f9a0640 -1 rbd::mirror::MirrorStatusUpdater 0x55bd3b5766c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-17T23:40:57.559 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi038.stderr:2024-09-17T23:40:57.557+0000 7f9b5c9ba640 -1 rbd::mirror::NamespaceReplayer: 0x55bd3bffc000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-17T23:40:57.559 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi038.stderr:2024-09-17T23:40:57.557+0000 7f9b569ae640 -1 rbd::mirror::MirrorStatusUpdater 0x55bd3b50f0e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-17T23:40:57.559 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi038.stderr:2024-09-17T23:40:57.557+0000 7f9b5d1bb640 -1 rbd::mirror::NamespaceReplayer: 0x55bd3bffc000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-17T23:40:57.570 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Stopped 2024-09-17T23:40:57.570 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-17T23:40:57.581 INFO:tasks.ceph.ceph_manager.cluster2:waiting for clean 2024-09-17T23:40:57.581 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-17T23:40:57.917 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:40:57.918 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2024-09-17T23:40:57.930 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":1582,"stamp":"2024-09-17T23:40:57.070478+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":274,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4499756,"kb_used_data":17716,"kb_used_omap":4,"kb_used_meta":4481275,"kb_avail":278615764,"statfs":{"total":289910292480,"available":285302542336,"internally_reserved":0,"allocated":18141184,"data_stored":10166570,"data_compressed":5998,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4785,"internal_metadata":4588825935},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536712312,"num_objects":-165,"num_object_clones":0,"num_object_copies":-330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-165,"num_whiteouts":0,"num_read":-138065,"num_read_kb":-39664143,"num_write":-641489,"num_write_kb":-233401396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-6,"num_bytes_recovered":-16777216,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001705"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1015,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-17T23:40:48.371771+0000","last_change":"2024-09-17T23:05:15.618717+0000","last_active":"2024-09-17T23:40:48.371771+0000","last_peered":"2024-09-17T23:40:48.371771+0000","last_clean":"2024-09-17T23:40:48.371771+0000","last_became_active":"2024-09-17T23:05:13.614964+0000","last_became_peered":"2024-09-17T23:05:13.614964+0000","last_unstale":"2024-09-17T23:40:48.371771+0000","last_undegraded":"2024-09-17T23:40:48.371771+0000","last_fullsized":"2024-09-17T23:40:48.371771+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:05:12.587919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:05:12.587919+0000","last_clean_scrub_stamp":"2024-09-17T23:05:12.587919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T02:02:11.663659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001157548,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1015,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-17T23:40:48.372046+0000","last_change":"2024-09-17T23:05:15.618037+0000","last_active":"2024-09-17T23:40:48.372046+0000","last_peered":"2024-09-17T23:40:48.372046+0000","last_clean":"2024-09-17T23:40:48.372046+0000","last_became_active":"2024-09-17T23:05:13.615179+0000","last_became_peered":"2024-09-17T23:05:13.615179+0000","last_unstale":"2024-09-17T23:40:48.372046+0000","last_undegraded":"2024-09-17T23:40:48.372046+0000","last_fullsized":"2024-09-17T23:40:48.372046+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:05:12.587919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:05:12.587919+0000","last_clean_scrub_stamp":"2024-09-17T23:05:12.587919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T07:27:12.449968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065459800000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1016,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-17T23:40:48.373617+0000","last_change":"2024-09-17T23:05:15.617997+0000","last_active":"2024-09-17T23:40:48.373617+0000","last_peered":"2024-09-17T23:40:48.373617+0000","last_clean":"2024-09-17T23:40:48.373617+0000","last_became_active":"2024-09-17T23:05:13.615245+0000","last_became_peered":"2024-09-17T23:05:13.615245+0000","last_unstale":"2024-09-17T23:40:48.373617+0000","last_undegraded":"2024-09-17T23:40:48.373617+0000","last_fullsized":"2024-09-17T23:40:48.373617+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:05:12.587919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:05:12.587919+0000","last_clean_scrub_stamp":"2024-09-17T23:05:12.587919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T05:02:26.092734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063722900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1016,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-17T23:40:48.370571+0000","last_change":"2024-09-17T23:05:15.618350+0000","last_active":"2024-09-17T23:40:48.370571+0000","last_peered":"2024-09-17T23:40:48.370571+0000","last_clean":"2024-09-17T23:40:48.370571+0000","last_became_active":"2024-09-17T23:05:13.615118+0000","last_became_peered":"2024-09-17T23:05:13.615118+0000","last_unstale":"2024-09-17T23:40:48.370571+0000","last_undegraded":"2024-09-17T23:40:48.370571+0000","last_fullsized":"2024-09-17T23:40:48.370571+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:05:12.587919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:05:12.587919+0000","last_clean_scrub_stamp":"2024-09-17T23:05:12.587919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T08:01:08.978654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097848599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"17'2","reported_seq":1018,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-17T23:40:48.343673+0000","last_change":"2024-09-17T23:05:15.612022+0000","last_active":"2024-09-17T23:40:48.343673+0000","last_peered":"2024-09-17T23:40:48.343673+0000","last_clean":"2024-09-17T23:40:48.343673+0000","last_became_active":"2024-09-17T23:05:13.615350+0000","last_became_peered":"2024-09-17T23:05:13.615350+0000","last_unstale":"2024-09-17T23:40:48.343673+0000","last_undegraded":"2024-09-17T23:40:48.343673+0000","last_fullsized":"2024-09-17T23:40:48.343673+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:05:12.587919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:05:12.587919+0000","last_clean_scrub_stamp":"2024-09-17T23:05:12.587919+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T00:22:33.153175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049918900000000003,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1016,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-17T23:40:48.334653+0000","last_change":"2024-09-17T23:05:16.398203+0000","last_active":"2024-09-17T23:40:48.334653+0000","last_peered":"2024-09-17T23:40:48.334653+0000","last_clean":"2024-09-17T23:40:48.334653+0000","last_became_active":"2024-09-17T23:05:13.615287+0000","last_became_peered":"2024-09-17T23:05:13.615287+0000","last_unstale":"2024-09-17T23:40:48.334653+0000","last_undegraded":"2024-09-17T23:40:48.334653+0000","last_fullsized":"2024-09-17T23:40:48.334653+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:05:12.587919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:05:12.587919+0000","last_clean_scrub_stamp":"2024-09-17T23:05:12.587919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T10:27:38.318149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031014299999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1016,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-17T23:40:48.333121+0000","last_change":"2024-09-17T23:05:16.398209+0000","last_active":"2024-09-17T23:40:48.333121+0000","last_peered":"2024-09-17T23:40:48.333121+0000","last_clean":"2024-09-17T23:40:48.333121+0000","last_became_active":"2024-09-17T23:05:13.615033+0000","last_became_peered":"2024-09-17T23:05:13.615033+0000","last_unstale":"2024-09-17T23:40:48.333121+0000","last_undegraded":"2024-09-17T23:40:48.333121+0000","last_fullsized":"2024-09-17T23:40:48.333121+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:05:12.587919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:05:12.587919+0000","last_clean_scrub_stamp":"2024-09-17T23:05:12.587919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T01:58:05.905647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031522499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"15'1","reported_seq":1017,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-17T23:40:48.373947+0000","last_change":"2024-09-17T23:05:15.618439+0000","last_active":"2024-09-17T23:40:48.373947+0000","last_peered":"2024-09-17T23:40:48.373947+0000","last_clean":"2024-09-17T23:40:48.373947+0000","last_became_active":"2024-09-17T23:05:13.609294+0000","last_became_peered":"2024-09-17T23:05:13.609294+0000","last_unstale":"2024-09-17T23:40:48.373947+0000","last_undegraded":"2024-09-17T23:40:48.373947+0000","last_fullsized":"2024-09-17T23:40:48.373947+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:05:12.587919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:05:12.587919+0000","last_clean_scrub_stamp":"2024-09-17T23:05:12.587919+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T02:57:25.565118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092743800000000005,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":1144,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-17T23:40:48.373830+0000","last_change":"2024-09-17T23:05:09.702570+0000","last_active":"2024-09-17T23:40:48.373830+0000","last_peered":"2024-09-17T23:40:48.373830+0000","last_clean":"2024-09-17T23:40:48.373830+0000","last_became_active":"2024-09-17T23:05:09.701419+0000","last_became_peered":"2024-09-17T23:05:09.701419+0000","last_unstale":"2024-09-17T23:40:48.373830+0000","last_undegraded":"2024-09-17T23:40:48.373830+0000","last_fullsized":"2024-09-17T23:40:48.373830+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:05:08.557542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:05:08.557542+0000","last_clean_scrub_stamp":"2024-09-17T23:05:08.557542+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T08:10:48.938230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":5998,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654706099,"num_pgs":83,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1431548,"kb_used_data":5868,"kb_used_omap":1,"kb_used_meta":1425662,"kb_avail":92940292,"statfs":{"total":96636764160,"available":95170859008,"internally_reserved":0,"allocated":6008832,"data_stored":2995278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1459878341},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706098,"num_pgs":97,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1382260,"kb_used_data":5924,"kb_used_omap":1,"kb_used_meta":1376318,"kb_avail":92989580,"statfs":{"total":96636764160,"available":95221329920,"internally_reserved":0,"allocated":6066176,"data_stored":3585646,"data_compressed":2999,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":1409350085},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706097,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1685948,"kb_used_data":5924,"kb_used_omap":1,"kb_used_meta":1679294,"kb_avail":92685892,"statfs":{"total":96636764160,"available":94910353408,"internally_reserved":0,"allocated":6066176,"data_stored":3585646,"data_compressed":2999,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":1719597509},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2999,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2999,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T23:40:57.931 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-17T23:40:58.219 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:40:58.219 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2024-09-17T23:40:58.231 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":1582,"stamp":"2024-09-17T23:40:57.070478+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":274,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4499756,"kb_used_data":17716,"kb_used_omap":4,"kb_used_meta":4481275,"kb_avail":278615764,"statfs":{"total":289910292480,"available":285302542336,"internally_reserved":0,"allocated":18141184,"data_stored":10166570,"data_compressed":5998,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4785,"internal_metadata":4588825935},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536712312,"num_objects":-165,"num_object_clones":0,"num_object_copies":-330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-165,"num_whiteouts":0,"num_read":-138065,"num_read_kb":-39664143,"num_write":-641489,"num_write_kb":-233401396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-6,"num_bytes_recovered":-16777216,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001705"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1015,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-17T23:40:48.371771+0000","last_change":"2024-09-17T23:05:15.618717+0000","last_active":"2024-09-17T23:40:48.371771+0000","last_peered":"2024-09-17T23:40:48.371771+0000","last_clean":"2024-09-17T23:40:48.371771+0000","last_became_active":"2024-09-17T23:05:13.614964+0000","last_became_peered":"2024-09-17T23:05:13.614964+0000","last_unstale":"2024-09-17T23:40:48.371771+0000","last_undegraded":"2024-09-17T23:40:48.371771+0000","last_fullsized":"2024-09-17T23:40:48.371771+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:05:12.587919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:05:12.587919+0000","last_clean_scrub_stamp":"2024-09-17T23:05:12.587919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T02:02:11.663659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001157548,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1015,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-17T23:40:48.372046+0000","last_change":"2024-09-17T23:05:15.618037+0000","last_active":"2024-09-17T23:40:48.372046+0000","last_peered":"2024-09-17T23:40:48.372046+0000","last_clean":"2024-09-17T23:40:48.372046+0000","last_became_active":"2024-09-17T23:05:13.615179+0000","last_became_peered":"2024-09-17T23:05:13.615179+0000","last_unstale":"2024-09-17T23:40:48.372046+0000","last_undegraded":"2024-09-17T23:40:48.372046+0000","last_fullsized":"2024-09-17T23:40:48.372046+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:05:12.587919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:05:12.587919+0000","last_clean_scrub_stamp":"2024-09-17T23:05:12.587919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T07:27:12.449968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065459800000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1016,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-17T23:40:48.373617+0000","last_change":"2024-09-17T23:05:15.617997+0000","last_active":"2024-09-17T23:40:48.373617+0000","last_peered":"2024-09-17T23:40:48.373617+0000","last_clean":"2024-09-17T23:40:48.373617+0000","last_became_active":"2024-09-17T23:05:13.615245+0000","last_became_peered":"2024-09-17T23:05:13.615245+0000","last_unstale":"2024-09-17T23:40:48.373617+0000","last_undegraded":"2024-09-17T23:40:48.373617+0000","last_fullsized":"2024-09-17T23:40:48.373617+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:05:12.587919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:05:12.587919+0000","last_clean_scrub_stamp":"2024-09-17T23:05:12.587919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T05:02:26.092734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063722900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1016,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-17T23:40:48.370571+0000","last_change":"2024-09-17T23:05:15.618350+0000","last_active":"2024-09-17T23:40:48.370571+0000","last_peered":"2024-09-17T23:40:48.370571+0000","last_clean":"2024-09-17T23:40:48.370571+0000","last_became_active":"2024-09-17T23:05:13.615118+0000","last_became_peered":"2024-09-17T23:05:13.615118+0000","last_unstale":"2024-09-17T23:40:48.370571+0000","last_undegraded":"2024-09-17T23:40:48.370571+0000","last_fullsized":"2024-09-17T23:40:48.370571+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:05:12.587919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:05:12.587919+0000","last_clean_scrub_stamp":"2024-09-17T23:05:12.587919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T08:01:08.978654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097848599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"17'2","reported_seq":1018,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-17T23:40:48.343673+0000","last_change":"2024-09-17T23:05:15.612022+0000","last_active":"2024-09-17T23:40:48.343673+0000","last_peered":"2024-09-17T23:40:48.343673+0000","last_clean":"2024-09-17T23:40:48.343673+0000","last_became_active":"2024-09-17T23:05:13.615350+0000","last_became_peered":"2024-09-17T23:05:13.615350+0000","last_unstale":"2024-09-17T23:40:48.343673+0000","last_undegraded":"2024-09-17T23:40:48.343673+0000","last_fullsized":"2024-09-17T23:40:48.343673+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:05:12.587919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:05:12.587919+0000","last_clean_scrub_stamp":"2024-09-17T23:05:12.587919+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T00:22:33.153175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049918900000000003,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1016,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-17T23:40:48.334653+0000","last_change":"2024-09-17T23:05:16.398203+0000","last_active":"2024-09-17T23:40:48.334653+0000","last_peered":"2024-09-17T23:40:48.334653+0000","last_clean":"2024-09-17T23:40:48.334653+0000","last_became_active":"2024-09-17T23:05:13.615287+0000","last_became_peered":"2024-09-17T23:05:13.615287+0000","last_unstale":"2024-09-17T23:40:48.334653+0000","last_undegraded":"2024-09-17T23:40:48.334653+0000","last_fullsized":"2024-09-17T23:40:48.334653+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:05:12.587919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:05:12.587919+0000","last_clean_scrub_stamp":"2024-09-17T23:05:12.587919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T10:27:38.318149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031014299999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1016,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-17T23:40:48.333121+0000","last_change":"2024-09-17T23:05:16.398209+0000","last_active":"2024-09-17T23:40:48.333121+0000","last_peered":"2024-09-17T23:40:48.333121+0000","last_clean":"2024-09-17T23:40:48.333121+0000","last_became_active":"2024-09-17T23:05:13.615033+0000","last_became_peered":"2024-09-17T23:05:13.615033+0000","last_unstale":"2024-09-17T23:40:48.333121+0000","last_undegraded":"2024-09-17T23:40:48.333121+0000","last_fullsized":"2024-09-17T23:40:48.333121+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:05:12.587919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:05:12.587919+0000","last_clean_scrub_stamp":"2024-09-17T23:05:12.587919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T01:58:05.905647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031522499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"15'1","reported_seq":1017,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-17T23:40:48.373947+0000","last_change":"2024-09-17T23:05:15.618439+0000","last_active":"2024-09-17T23:40:48.373947+0000","last_peered":"2024-09-17T23:40:48.373947+0000","last_clean":"2024-09-17T23:40:48.373947+0000","last_became_active":"2024-09-17T23:05:13.609294+0000","last_became_peered":"2024-09-17T23:05:13.609294+0000","last_unstale":"2024-09-17T23:40:48.373947+0000","last_undegraded":"2024-09-17T23:40:48.373947+0000","last_fullsized":"2024-09-17T23:40:48.373947+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:05:12.587919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:05:12.587919+0000","last_clean_scrub_stamp":"2024-09-17T23:05:12.587919+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T02:57:25.565118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092743800000000005,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":1144,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-17T23:40:48.373830+0000","last_change":"2024-09-17T23:05:09.702570+0000","last_active":"2024-09-17T23:40:48.373830+0000","last_peered":"2024-09-17T23:40:48.373830+0000","last_clean":"2024-09-17T23:40:48.373830+0000","last_became_active":"2024-09-17T23:05:09.701419+0000","last_became_peered":"2024-09-17T23:05:09.701419+0000","last_unstale":"2024-09-17T23:40:48.373830+0000","last_undegraded":"2024-09-17T23:40:48.373830+0000","last_fullsized":"2024-09-17T23:40:48.373830+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:05:08.557542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:05:08.557542+0000","last_clean_scrub_stamp":"2024-09-17T23:05:08.557542+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T08:10:48.938230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":5998,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654706099,"num_pgs":83,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1431548,"kb_used_data":5868,"kb_used_omap":1,"kb_used_meta":1425662,"kb_avail":92940292,"statfs":{"total":96636764160,"available":95170859008,"internally_reserved":0,"allocated":6008832,"data_stored":2995278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1459878341},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706098,"num_pgs":97,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1382260,"kb_used_data":5924,"kb_used_omap":1,"kb_used_meta":1376318,"kb_avail":92989580,"statfs":{"total":96636764160,"available":95221329920,"internally_reserved":0,"allocated":6066176,"data_stored":3585646,"data_compressed":2999,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":1409350085},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706097,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1685948,"kb_used_data":5924,"kb_used_omap":1,"kb_used_meta":1679294,"kb_avail":92685892,"statfs":{"total":96636764160,"available":94910353408,"internally_reserved":0,"allocated":6066176,"data_stored":3585646,"data_compressed":2999,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":1719597509},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2999,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2999,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T23:40:58.232 INFO:tasks.ceph.ceph_manager.cluster2:clean! 2024-09-17T23:40:58.232 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-17T23:40:58.518 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:40:58.518 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2024-09-17T23:40:58.530 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":1582,"stamp":"2024-09-17T23:40:57.070478+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":274,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4499756,"kb_used_data":17716,"kb_used_omap":4,"kb_used_meta":4481275,"kb_avail":278615764,"statfs":{"total":289910292480,"available":285302542336,"internally_reserved":0,"allocated":18141184,"data_stored":10166570,"data_compressed":5998,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4785,"internal_metadata":4588825935},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536712312,"num_objects":-165,"num_object_clones":0,"num_object_copies":-330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-165,"num_whiteouts":0,"num_read":-138065,"num_read_kb":-39664143,"num_write":-641489,"num_write_kb":-233401396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-6,"num_bytes_recovered":-16777216,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001705"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1015,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-17T23:40:48.371771+0000","last_change":"2024-09-17T23:05:15.618717+0000","last_active":"2024-09-17T23:40:48.371771+0000","last_peered":"2024-09-17T23:40:48.371771+0000","last_clean":"2024-09-17T23:40:48.371771+0000","last_became_active":"2024-09-17T23:05:13.614964+0000","last_became_peered":"2024-09-17T23:05:13.614964+0000","last_unstale":"2024-09-17T23:40:48.371771+0000","last_undegraded":"2024-09-17T23:40:48.371771+0000","last_fullsized":"2024-09-17T23:40:48.371771+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:05:12.587919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:05:12.587919+0000","last_clean_scrub_stamp":"2024-09-17T23:05:12.587919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T02:02:11.663659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001157548,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1015,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-17T23:40:48.372046+0000","last_change":"2024-09-17T23:05:15.618037+0000","last_active":"2024-09-17T23:40:48.372046+0000","last_peered":"2024-09-17T23:40:48.372046+0000","last_clean":"2024-09-17T23:40:48.372046+0000","last_became_active":"2024-09-17T23:05:13.615179+0000","last_became_peered":"2024-09-17T23:05:13.615179+0000","last_unstale":"2024-09-17T23:40:48.372046+0000","last_undegraded":"2024-09-17T23:40:48.372046+0000","last_fullsized":"2024-09-17T23:40:48.372046+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:05:12.587919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:05:12.587919+0000","last_clean_scrub_stamp":"2024-09-17T23:05:12.587919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T07:27:12.449968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065459800000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1016,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-17T23:40:48.373617+0000","last_change":"2024-09-17T23:05:15.617997+0000","last_active":"2024-09-17T23:40:48.373617+0000","last_peered":"2024-09-17T23:40:48.373617+0000","last_clean":"2024-09-17T23:40:48.373617+0000","last_became_active":"2024-09-17T23:05:13.615245+0000","last_became_peered":"2024-09-17T23:05:13.615245+0000","last_unstale":"2024-09-17T23:40:48.373617+0000","last_undegraded":"2024-09-17T23:40:48.373617+0000","last_fullsized":"2024-09-17T23:40:48.373617+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:05:12.587919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:05:12.587919+0000","last_clean_scrub_stamp":"2024-09-17T23:05:12.587919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T05:02:26.092734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063722900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1016,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-17T23:40:48.370571+0000","last_change":"2024-09-17T23:05:15.618350+0000","last_active":"2024-09-17T23:40:48.370571+0000","last_peered":"2024-09-17T23:40:48.370571+0000","last_clean":"2024-09-17T23:40:48.370571+0000","last_became_active":"2024-09-17T23:05:13.615118+0000","last_became_peered":"2024-09-17T23:05:13.615118+0000","last_unstale":"2024-09-17T23:40:48.370571+0000","last_undegraded":"2024-09-17T23:40:48.370571+0000","last_fullsized":"2024-09-17T23:40:48.370571+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:05:12.587919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:05:12.587919+0000","last_clean_scrub_stamp":"2024-09-17T23:05:12.587919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T08:01:08.978654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097848599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"17'2","reported_seq":1018,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-17T23:40:48.343673+0000","last_change":"2024-09-17T23:05:15.612022+0000","last_active":"2024-09-17T23:40:48.343673+0000","last_peered":"2024-09-17T23:40:48.343673+0000","last_clean":"2024-09-17T23:40:48.343673+0000","last_became_active":"2024-09-17T23:05:13.615350+0000","last_became_peered":"2024-09-17T23:05:13.615350+0000","last_unstale":"2024-09-17T23:40:48.343673+0000","last_undegraded":"2024-09-17T23:40:48.343673+0000","last_fullsized":"2024-09-17T23:40:48.343673+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:05:12.587919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:05:12.587919+0000","last_clean_scrub_stamp":"2024-09-17T23:05:12.587919+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T00:22:33.153175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049918900000000003,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1016,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-17T23:40:48.334653+0000","last_change":"2024-09-17T23:05:16.398203+0000","last_active":"2024-09-17T23:40:48.334653+0000","last_peered":"2024-09-17T23:40:48.334653+0000","last_clean":"2024-09-17T23:40:48.334653+0000","last_became_active":"2024-09-17T23:05:13.615287+0000","last_became_peered":"2024-09-17T23:05:13.615287+0000","last_unstale":"2024-09-17T23:40:48.334653+0000","last_undegraded":"2024-09-17T23:40:48.334653+0000","last_fullsized":"2024-09-17T23:40:48.334653+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:05:12.587919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:05:12.587919+0000","last_clean_scrub_stamp":"2024-09-17T23:05:12.587919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T10:27:38.318149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031014299999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1016,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-17T23:40:48.333121+0000","last_change":"2024-09-17T23:05:16.398209+0000","last_active":"2024-09-17T23:40:48.333121+0000","last_peered":"2024-09-17T23:40:48.333121+0000","last_clean":"2024-09-17T23:40:48.333121+0000","last_became_active":"2024-09-17T23:05:13.615033+0000","last_became_peered":"2024-09-17T23:05:13.615033+0000","last_unstale":"2024-09-17T23:40:48.333121+0000","last_undegraded":"2024-09-17T23:40:48.333121+0000","last_fullsized":"2024-09-17T23:40:48.333121+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:05:12.587919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:05:12.587919+0000","last_clean_scrub_stamp":"2024-09-17T23:05:12.587919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T01:58:05.905647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031522499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"15'1","reported_seq":1017,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-17T23:40:48.373947+0000","last_change":"2024-09-17T23:05:15.618439+0000","last_active":"2024-09-17T23:40:48.373947+0000","last_peered":"2024-09-17T23:40:48.373947+0000","last_clean":"2024-09-17T23:40:48.373947+0000","last_became_active":"2024-09-17T23:05:13.609294+0000","last_became_peered":"2024-09-17T23:05:13.609294+0000","last_unstale":"2024-09-17T23:40:48.373947+0000","last_undegraded":"2024-09-17T23:40:48.373947+0000","last_fullsized":"2024-09-17T23:40:48.373947+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:05:12.587919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:05:12.587919+0000","last_clean_scrub_stamp":"2024-09-17T23:05:12.587919+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T02:57:25.565118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092743800000000005,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":1144,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-17T23:40:48.373830+0000","last_change":"2024-09-17T23:05:09.702570+0000","last_active":"2024-09-17T23:40:48.373830+0000","last_peered":"2024-09-17T23:40:48.373830+0000","last_clean":"2024-09-17T23:40:48.373830+0000","last_became_active":"2024-09-17T23:05:09.701419+0000","last_became_peered":"2024-09-17T23:05:09.701419+0000","last_unstale":"2024-09-17T23:40:48.373830+0000","last_undegraded":"2024-09-17T23:40:48.373830+0000","last_fullsized":"2024-09-17T23:40:48.373830+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:05:08.557542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:05:08.557542+0000","last_clean_scrub_stamp":"2024-09-17T23:05:08.557542+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T08:10:48.938230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":5998,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654706099,"num_pgs":83,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1431548,"kb_used_data":5868,"kb_used_omap":1,"kb_used_meta":1425662,"kb_avail":92940292,"statfs":{"total":96636764160,"available":95170859008,"internally_reserved":0,"allocated":6008832,"data_stored":2995278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1459878341},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706098,"num_pgs":97,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1382260,"kb_used_data":5924,"kb_used_omap":1,"kb_used_meta":1376318,"kb_avail":92989580,"statfs":{"total":96636764160,"available":95221329920,"internally_reserved":0,"allocated":6066176,"data_stored":3585646,"data_compressed":2999,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":1409350085},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706097,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1685948,"kb_used_data":5924,"kb_used_omap":1,"kb_used_meta":1679294,"kb_avail":92685892,"statfs":{"total":96636764160,"available":94910353408,"internally_reserved":0,"allocated":6066176,"data_stored":3585646,"data_compressed":2999,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":1719597509},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2999,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2999,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T23:40:58.531 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-09-17T23:40:58.823 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:40:58.823 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":509,"fsid":"9e14216a-77a4-4aa5-9521-b021dc9cf0cd","created":"2024-09-17T23:04:56.974990+0000","modified":"2024-09-17T23:40:56.364551+0000","last_up_change":"2024-09-17T23:05:06.530460+0000","last_in_change":"2024-09-17T23:04:57.825666+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T23:05:08.551128+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-17T23:05:12.248805+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":17,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"411d4107-4e3a-4b68-a515-eafd1b3b3b4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":1291026539},{"type":"v1","addr":"172.21.15.38:6817","nonce":1291026539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":1291026539},{"type":"v1","addr":"172.21.15.38:6819","nonce":1291026539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":1291026539},{"type":"v1","addr":"172.21.15.38:6823","nonce":1291026539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":1291026539},{"type":"v1","addr":"172.21.15.38:6821","nonce":1291026539}]},"public_addr":"172.21.15.38:6817/1291026539","cluster_addr":"172.21.15.38:6819/1291026539","heartbeat_back_addr":"172.21.15.38:6823/1291026539","heartbeat_front_addr":"172.21.15.38:6821/1291026539","state":["exists","up"]},{"osd":1,"uuid":"7a899e9e-c234-4128-8b3c-5234e7a66aed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":2141962341},{"type":"v1","addr":"172.21.15.38:6809","nonce":2141962341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":2141962341},{"type":"v1","addr":"172.21.15.38:6811","nonce":2141962341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":2141962341},{"type":"v1","addr":"172.21.15.38:6815","nonce":2141962341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2141962341},{"type":"v1","addr":"172.21.15.38:6813","nonce":2141962341}]},"public_addr":"172.21.15.38:6809/2141962341","cluster_addr":"172.21.15.38:6811/2141962341","heartbeat_back_addr":"172.21.15.38:6815/2141962341","heartbeat_front_addr":"172.21.15.38:6813/2141962341","state":["exists","up"]},{"osd":2,"uuid":"06a70fc4-b466-44d3-9800-2e0c9780993e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":2430931791},{"type":"v1","addr":"172.21.15.38:6801","nonce":2430931791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":2430931791},{"type":"v1","addr":"172.21.15.38:6803","nonce":2430931791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2430931791},{"type":"v1","addr":"172.21.15.38:6807","nonce":2430931791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":2430931791},{"type":"v1","addr":"172.21.15.38:6805","nonce":2430931791}]},"public_addr":"172.21.15.38:6801/2430931791","cluster_addr":"172.21.15.38:6803/2430931791","heartbeat_back_addr":"172.21.15.38:6807/2430931791","heartbeat_front_addr":"172.21.15.38:6805/2430931791","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T23:05:04.623873+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T23:05:04.361551+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T23:05:04.333856+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.38:0/1417776384":"2024-09-18T00:40:46.725938+0000","172.21.15.38:0/1879343591":"2024-09-18T00:38:49.069204+0000","172.21.15.38:0/449581764":"2024-09-18T00:38:12.352923+0000","172.21.15.38:0/2701017058":"2024-09-18T00:37:29.776995+0000","172.21.15.38:0/3075968307":"2024-09-18T00:36:46.010158+0000","172.21.15.38:0/2677739130":"2024-09-18T00:34:34.829275+0000","172.21.15.38:0/2976288844":"2024-09-18T00:31:04.383701+0000","172.21.15.38:0/4282399984":"2024-09-18T00:27:08.219766+0000","172.21.15.38:0/2841967229":"2024-09-18T00:14:18.099920+0000","172.21.15.38:0/1528758251":"2024-09-18T00:26:38.370572+0000","172.21.15.38:0/1172804606":"2024-09-18T00:26:52.486945+0000","172.21.15.38:0/2289770168":"2024-09-18T00:26:31.356772+0000","172.21.15.38:0/3895362932":"2024-09-18T00:27:22.259513+0000","172.21.15.38:0/3802907733":"2024-09-18T00:25:26.115028+0000","172.21.15.38:0/1723092101":"2024-09-18T00:26:17.319895+0000","172.21.15.38:0/1988761749":"2024-09-18T00:28:04.449818+0000","172.21.15.38:0/1694911845":"2024-09-18T00:26:10.285148+0000","172.21.15.38:0/110985380":"2024-09-18T00:39:27.942579+0000","172.21.15.38:0/1321484832":"2024-09-18T00:25:56.186385+0000","172.21.15.38:0/1954513777":"2024-09-18T00:30:43.240718+0000","172.21.15.38:0/206938225":"2024-09-18T00:25:34.334866+0000","172.21.15.38:0/236173734":"2024-09-18T00:27:15.644415+0000","172.21.15.38:0/1322653118":"2024-09-18T00:13:32.072750+0000","172.21.15.38:0/4000441206":"2024-09-18T00:29:59.352916+0000","172.21.15.38:0/400118576":"2024-09-18T00:40:06.215474+0000","172.21.15.38:0/3995413952":"2024-09-18T00:08:31.732189+0000","172.21.15.38:0/2377733228":"2024-09-18T00:25:03.365066+0000","172.21.15.38:0/1822345633":"2024-09-18T00:27:43.334213+0000","172.21.15.38:0/4174635406":"2024-09-18T00:30:50.252619+0000","172.21.15.38:0/411269487":"2024-09-18T00:09:26.743540+0000","172.21.15.38:0/999704036":"2024-09-18T00:30:27.525373+0000","172.21.15.38:0/2061454822":"2024-09-18T00:36:00.815875+0000","172.21.15.38:0/1286749872":"2024-09-18T00:11:04.338518+0000","172.21.15.38:0/2877981501":"2024-09-18T00:25:49.116468+0000","172.21.15.38:0/2581385132":"2024-09-18T00:15:05.842858+0000","172.21.15.38:0/3719424694":"2024-09-18T00:11:56.642209+0000","172.21.15.38:0/4204289966":"2024-09-18T00:28:27.444532+0000","172.21.15.38:0/2726715570":"2024-09-18T00:12:44.069541+0000","172.21.15.38:0/774236260":"2024-09-18T00:07:41.321883+0000","172.21.15.38:0/3308995597":"2024-09-18T00:27:36.335646+0000","172.21.15.38:0/754137504":"2024-09-18T00:25:42.290339+0000","172.21.15.38:0/656615720":"2024-09-18T00:26:24.536321+0000","172.21.15.38:0/2445679546":"2024-09-18T00:28:48.542174+0000","172.21.15.38:0/176515456":"2024-09-18T00:25:17.453072+0000","172.21.15.38:0/628260083":"2024-09-18T00:25:10.420781+0000","172.21.15.38:0/2913179775":"2024-09-18T00:27:29.240398+0000","172.21.15.38:0/2514513504":"2024-09-18T00:30:06.384674+0000","172.21.15.38:0/701816208":"2024-09-18T00:26:59.469713+0000","172.21.15.38:0/2202901687":"2024-09-18T00:27:50.360281+0000","172.21.15.38:0/2356933808":"2024-09-18T00:27:57.562100+0000","172.21.15.38:0/77968017":"2024-09-18T00:28:34.267572+0000","172.21.15.38:0/410158318":"2024-09-18T00:30:36.197964+0000","172.21.15.38:0/2350830971":"2024-09-18T00:28:20.178432+0000","172.21.15.38:0/1158246531":"2024-09-18T00:29:02.404115+0000","172.21.15.38:0/1078961985":"2024-09-18T00:29:27.200147+0000","172.21.15.38:0/2190652114":"2024-09-18T00:28:41.300436+0000","172.21.15.38:0/3931803663":"2024-09-18T00:28:55.378933+0000","172.21.15.38:0/3341330855":"2024-09-18T00:29:09.448585+0000","172.21.15.38:0/4112448817":"2024-09-18T00:10:16.123358+0000","172.21.15.38:0/2460230195":"2024-09-18T00:29:18.319290+0000","172.21.15.38:0/224120635":"2024-09-18T00:29:41.445760+0000","172.21.15.38:0/3924011598":"2024-09-18T00:26:03.222651+0000","172.21.15.38:0/4067245520":"2024-09-18T00:29:34.255447+0000","172.21.15.38:0/108180290":"2024-09-18T00:35:16.744638+0000","172.21.15.38:0/1187459337":"2024-09-18T00:30:13.605911+0000","172.21.15.38:0/4075529033":"2024-09-18T00:26:45.419145+0000","172.21.15.38:0/1927024722":"2024-09-18T00:29:52.299983+0000","172.21.15.38:0/4021224207":"2024-09-18T00:28:11.510497+0000","172.21.15.38:0/2493761348":"2024-09-18T00:30:20.435756+0000","172.21.15.38:0/201516797":"2024-09-18T00:30:57.348148+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":194,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T23:40:59.836 INFO:tasks.ceph:Scrubbing cluster2.osd.0 2024-09-17T23:40:59.836 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-17T23:40:59.977 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-09-17T23:40:59.977 INFO:teuthology.orchestra.run.smithi038.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-17T23:40:59.977 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-09-17T23:40:59.988 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd deep-scrub 0 2024-09-17T23:41:00.276 INFO:teuthology.orchestra.run.smithi038.stderr:instructed osd(s) 0 to deep-scrub 2024-09-17T23:41:00.289 INFO:tasks.ceph:Scrubbing cluster2.osd.1 2024-09-17T23:41:00.289 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-17T23:41:00.416 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-09-17T23:41:00.416 INFO:teuthology.orchestra.run.smithi038.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-17T23:41:00.416 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-09-17T23:41:00.426 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd deep-scrub 1 2024-09-17T23:41:00.718 INFO:teuthology.orchestra.run.smithi038.stderr:instructed osd(s) 1 to deep-scrub 2024-09-17T23:41:00.731 INFO:tasks.ceph:Scrubbing cluster2.osd.2 2024-09-17T23:41:00.731 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-17T23:41:00.857 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-09-17T23:41:00.857 INFO:teuthology.orchestra.run.smithi038.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-17T23:41:00.857 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-09-17T23:41:00.868 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd deep-scrub 2 2024-09-17T23:41:01.161 INFO:teuthology.orchestra.run.smithi038.stderr:instructed osd(s) 2 to deep-scrub 2024-09-17T23:41:01.174 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-17T23:41:01.462 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:41:01.463 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2024-09-17T23:41:01.475 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":1584,"stamp":"2024-09-17T23:41:01.071332+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":186,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4535204,"kb_used_data":17772,"kb_used_omap":4,"kb_used_meta":4517371,"kb_avail":278580316,"statfs":{"total":289910292480,"available":285266243584,"internally_reserved":0,"allocated":18198528,"data_stored":10194626,"data_compressed":5998,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4785,"internal_metadata":4625788239},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-38,"num_objects":-34,"num_object_clones":0,"num_object_copies":-68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-34,"num_whiteouts":0,"num_read":-138371,"num_read_kb":-39664406,"num_write":-641677,"num_write_kb":-233401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-6,"num_bytes_recovered":-16777216,"num_keys_recovered":0,"num_objects_omap":-13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001815"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1017,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-09-17T23:40:55.715143+0000","last_change":"2024-09-17T23:05:15.618717+0000","last_active":"2024-09-17T23:40:55.715143+0000","last_peered":"2024-09-17T23:40:55.715143+0000","last_clean":"2024-09-17T23:40:55.715143+0000","last_became_active":"2024-09-17T23:05:13.614964+0000","last_became_peered":"2024-09-17T23:05:13.614964+0000","last_unstale":"2024-09-17T23:40:55.715143+0000","last_undegraded":"2024-09-17T23:40:55.715143+0000","last_fullsized":"2024-09-17T23:40:55.715143+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:05:12.587919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:05:12.587919+0000","last_clean_scrub_stamp":"2024-09-17T23:05:12.587919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T02:02:11.663659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001157548,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1017,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-09-17T23:40:55.714617+0000","last_change":"2024-09-17T23:05:15.618037+0000","last_active":"2024-09-17T23:40:55.714617+0000","last_peered":"2024-09-17T23:40:55.714617+0000","last_clean":"2024-09-17T23:40:55.714617+0000","last_became_active":"2024-09-17T23:05:13.615179+0000","last_became_peered":"2024-09-17T23:05:13.615179+0000","last_unstale":"2024-09-17T23:40:55.714617+0000","last_undegraded":"2024-09-17T23:40:55.714617+0000","last_fullsized":"2024-09-17T23:40:55.714617+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:05:12.587919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:05:12.587919+0000","last_clean_scrub_stamp":"2024-09-17T23:05:12.587919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T07:27:12.449968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065459800000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1018,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-09-17T23:40:55.713007+0000","last_change":"2024-09-17T23:05:15.617997+0000","last_active":"2024-09-17T23:40:55.713007+0000","last_peered":"2024-09-17T23:40:55.713007+0000","last_clean":"2024-09-17T23:40:55.713007+0000","last_became_active":"2024-09-17T23:05:13.615245+0000","last_became_peered":"2024-09-17T23:05:13.615245+0000","last_unstale":"2024-09-17T23:40:55.713007+0000","last_undegraded":"2024-09-17T23:40:55.713007+0000","last_fullsized":"2024-09-17T23:40:55.713007+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:05:12.587919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:05:12.587919+0000","last_clean_scrub_stamp":"2024-09-17T23:05:12.587919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T05:02:26.092734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063722900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1018,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-09-17T23:40:55.711506+0000","last_change":"2024-09-17T23:05:15.618350+0000","last_active":"2024-09-17T23:40:55.711506+0000","last_peered":"2024-09-17T23:40:55.711506+0000","last_clean":"2024-09-17T23:40:55.711506+0000","last_became_active":"2024-09-17T23:05:13.615118+0000","last_became_peered":"2024-09-17T23:05:13.615118+0000","last_unstale":"2024-09-17T23:40:55.711506+0000","last_undegraded":"2024-09-17T23:40:55.711506+0000","last_fullsized":"2024-09-17T23:40:55.711506+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:05:12.587919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:05:12.587919+0000","last_clean_scrub_stamp":"2024-09-17T23:05:12.587919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T08:01:08.978654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097848599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"17'2","reported_seq":1022,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-17T23:40:56.372133+0000","last_change":"2024-09-17T23:05:15.612022+0000","last_active":"2024-09-17T23:40:56.372133+0000","last_peered":"2024-09-17T23:40:56.372133+0000","last_clean":"2024-09-17T23:40:56.372133+0000","last_became_active":"2024-09-17T23:05:13.615350+0000","last_became_peered":"2024-09-17T23:05:13.615350+0000","last_unstale":"2024-09-17T23:40:56.372133+0000","last_undegraded":"2024-09-17T23:40:56.372133+0000","last_fullsized":"2024-09-17T23:40:56.372133+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:05:12.587919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:05:12.587919+0000","last_clean_scrub_stamp":"2024-09-17T23:05:12.587919+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T00:22:33.153175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049918900000000003,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1020,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-17T23:40:57.708088+0000","last_change":"2024-09-17T23:05:16.398203+0000","last_active":"2024-09-17T23:40:57.708088+0000","last_peered":"2024-09-17T23:40:57.708088+0000","last_clean":"2024-09-17T23:40:57.708088+0000","last_became_active":"2024-09-17T23:05:13.615287+0000","last_became_peered":"2024-09-17T23:05:13.615287+0000","last_unstale":"2024-09-17T23:40:57.708088+0000","last_undegraded":"2024-09-17T23:40:57.708088+0000","last_fullsized":"2024-09-17T23:40:57.708088+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:05:12.587919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:05:12.587919+0000","last_clean_scrub_stamp":"2024-09-17T23:05:12.587919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T10:27:38.318149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031014299999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1020,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-17T23:40:57.703819+0000","last_change":"2024-09-17T23:05:16.398209+0000","last_active":"2024-09-17T23:40:57.703819+0000","last_peered":"2024-09-17T23:40:57.703819+0000","last_clean":"2024-09-17T23:40:57.703819+0000","last_became_active":"2024-09-17T23:05:13.615033+0000","last_became_peered":"2024-09-17T23:05:13.615033+0000","last_unstale":"2024-09-17T23:40:57.703819+0000","last_undegraded":"2024-09-17T23:40:57.703819+0000","last_fullsized":"2024-09-17T23:40:57.703819+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:05:12.587919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:05:12.587919+0000","last_clean_scrub_stamp":"2024-09-17T23:05:12.587919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T01:58:05.905647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031522499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"15'1","reported_seq":1019,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-09-17T23:40:55.712415+0000","last_change":"2024-09-17T23:05:15.618439+0000","last_active":"2024-09-17T23:40:55.712415+0000","last_peered":"2024-09-17T23:40:55.712415+0000","last_clean":"2024-09-17T23:40:55.712415+0000","last_became_active":"2024-09-17T23:05:13.609294+0000","last_became_peered":"2024-09-17T23:05:13.609294+0000","last_unstale":"2024-09-17T23:40:55.712415+0000","last_undegraded":"2024-09-17T23:40:55.712415+0000","last_fullsized":"2024-09-17T23:40:55.712415+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:05:12.587919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:05:12.587919+0000","last_clean_scrub_stamp":"2024-09-17T23:05:12.587919+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T02:57:25.565118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092743800000000005,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":1146,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-09-17T23:40:55.710582+0000","last_change":"2024-09-17T23:05:09.702570+0000","last_active":"2024-09-17T23:40:55.710582+0000","last_peered":"2024-09-17T23:40:55.710582+0000","last_clean":"2024-09-17T23:40:55.710582+0000","last_became_active":"2024-09-17T23:05:09.701419+0000","last_became_peered":"2024-09-17T23:05:09.701419+0000","last_unstale":"2024-09-17T23:40:55.710582+0000","last_undegraded":"2024-09-17T23:40:55.710582+0000","last_fullsized":"2024-09-17T23:40:55.710582+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:05:08.557542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:05:08.557542+0000","last_clean_scrub_stamp":"2024-09-17T23:05:08.557542+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T08:10:48.938230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":5998,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654706100,"num_pgs":83,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1449608,"kb_used_data":5880,"kb_used_omap":1,"kb_used_meta":1443710,"kb_avail":92922232,"statfs":{"total":96636764160,"available":95152365568,"internally_reserved":0,"allocated":6021120,"data_stored":3002401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1478359493},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706099,"num_pgs":97,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1400324,"kb_used_data":5940,"kb_used_omap":1,"kb_used_meta":1394366,"kb_avail":92971516,"statfs":{"total":96636764160,"available":95202832384,"internally_reserved":0,"allocated":6082560,"data_stored":3592788,"data_compressed":2999,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":1427831237},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706098,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1685272,"kb_used_data":5952,"kb_used_omap":1,"kb_used_meta":1679294,"kb_avail":92686568,"statfs":{"total":96636764160,"available":94911045632,"internally_reserved":0,"allocated":6094848,"data_stored":3599437,"data_compressed":2999,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":1719597509},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2999,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2999,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T23:41:01.478 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-17T23:05:12.587919+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=23, tm_min=5, tm_sec=12, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=23, tm_min=40, tm_sec=58, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T23:41:01.479 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-09-17T23:05:12.587919+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=23, tm_min=5, tm_sec=12, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=23, tm_min=40, tm_sec=58, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T23:41:01.479 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-17T23:05:12.587919+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=23, tm_min=5, tm_sec=12, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=23, tm_min=40, tm_sec=58, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T23:41:01.479 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-09-17T23:05:12.587919+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=23, tm_min=5, tm_sec=12, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=23, tm_min=40, tm_sec=58, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T23:41:01.479 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-09-17T23:05:12.587919+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=23, tm_min=5, tm_sec=12, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=23, tm_min=40, tm_sec=58, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T23:41:01.479 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-09-17T23:05:12.587919+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=23, tm_min=5, tm_sec=12, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=23, tm_min=40, tm_sec=58, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T23:41:01.479 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-17T23:05:12.587919+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=23, tm_min=5, tm_sec=12, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=23, tm_min=40, tm_sec=58, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T23:41:01.479 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-17T23:05:12.587919+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=23, tm_min=5, tm_sec=12, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=23, tm_min=40, tm_sec=58, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T23:41:01.479 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-17T23:05:08.557542+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=23, tm_min=5, tm_sec=8, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=23, tm_min=40, tm_sec=58, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T23:41:01.479 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-17T23:41:21.481 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-17T23:41:21.784 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-17T23:41:21.784 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2024-09-17T23:41:21.795 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":1594,"stamp":"2024-09-17T23:41:21.075211+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":104,"ondisk_log_size":104,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4535220,"kb_used_data":17796,"kb_used_omap":4,"kb_used_meta":4517371,"kb_avail":278580300,"statfs":{"total":289910292480,"available":285266227200,"internally_reserved":0,"allocated":18223104,"data_stored":10207924,"data_compressed":5998,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4785,"internal_metadata":4625788239},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002458"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1028,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-17T23:41:03.956428+0000","last_change":"2024-09-17T23:41:03.956452+0000","last_active":"2024-09-17T23:41:03.956428+0000","last_peered":"2024-09-17T23:41:03.956428+0000","last_clean":"2024-09-17T23:41:03.956428+0000","last_became_active":"2024-09-17T23:05:13.614964+0000","last_became_peered":"2024-09-17T23:05:13.614964+0000","last_unstale":"2024-09-17T23:41:03.956428+0000","last_undegraded":"2024-09-17T23:41:03.956428+0000","last_fullsized":"2024-09-17T23:41:03.956428+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:41:03.956326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:41:03.956326+0000","last_clean_scrub_stamp":"2024-09-17T23:41:03.956326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T00:06:02.794284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001157548,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1028,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-17T23:41:02.959237+0000","last_change":"2024-09-17T23:41:02.959263+0000","last_active":"2024-09-17T23:41:02.959237+0000","last_peered":"2024-09-17T23:41:02.959237+0000","last_clean":"2024-09-17T23:41:02.959237+0000","last_became_active":"2024-09-17T23:05:13.615179+0000","last_became_peered":"2024-09-17T23:05:13.615179+0000","last_unstale":"2024-09-17T23:41:02.959237+0000","last_undegraded":"2024-09-17T23:41:02.959237+0000","last_fullsized":"2024-09-17T23:41:02.959237+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:41:02.959130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:41:02.959130+0000","last_clean_scrub_stamp":"2024-09-17T23:41:02.959130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T11:14:44.131684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065459800000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1029,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-17T23:41:05.908197+0000","last_change":"2024-09-17T23:41:05.908259+0000","last_active":"2024-09-17T23:41:05.908197+0000","last_peered":"2024-09-17T23:41:05.908197+0000","last_clean":"2024-09-17T23:41:05.908197+0000","last_became_active":"2024-09-17T23:05:13.615245+0000","last_became_peered":"2024-09-17T23:05:13.615245+0000","last_unstale":"2024-09-17T23:41:05.908197+0000","last_undegraded":"2024-09-17T23:41:05.908197+0000","last_fullsized":"2024-09-17T23:41:05.908197+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:41:05.908083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:41:05.908083+0000","last_clean_scrub_stamp":"2024-09-17T23:41:05.908083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T04:08:41.037362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063722900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1029,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-17T23:41:04.924211+0000","last_change":"2024-09-17T23:41:04.924247+0000","last_active":"2024-09-17T23:41:04.924211+0000","last_peered":"2024-09-17T23:41:04.924211+0000","last_clean":"2024-09-17T23:41:04.924211+0000","last_became_active":"2024-09-17T23:05:13.615118+0000","last_became_peered":"2024-09-17T23:05:13.615118+0000","last_unstale":"2024-09-17T23:41:04.924211+0000","last_undegraded":"2024-09-17T23:41:04.924211+0000","last_fullsized":"2024-09-17T23:41:04.924211+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:41:04.924120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:41:04.924120+0000","last_clean_scrub_stamp":"2024-09-17T23:41:04.924120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T08:05:38.569001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097848599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"509'3","reported_seq":1032,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-17T23:41:00.388990+0000","last_change":"2024-09-17T23:41:00.389046+0000","last_active":"2024-09-17T23:41:00.388990+0000","last_peered":"2024-09-17T23:41:00.388990+0000","last_clean":"2024-09-17T23:41:00.388990+0000","last_became_active":"2024-09-17T23:05:13.615350+0000","last_became_peered":"2024-09-17T23:05:13.615350+0000","last_unstale":"2024-09-17T23:41:00.388990+0000","last_undegraded":"2024-09-17T23:41:00.388990+0000","last_fullsized":"2024-09-17T23:41:00.388990+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"509'3","last_scrub_stamp":"2024-09-17T23:41:00.388871+0000","last_deep_scrub":"509'3","last_deep_scrub_stamp":"2024-09-17T23:41:00.388871+0000","last_clean_scrub_stamp":"2024-09-17T23:41:00.388871+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T04:35:56.261655+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00049918900000000003,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1029,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-17T23:41:01.497887+0000","last_change":"2024-09-17T23:41:01.497915+0000","last_active":"2024-09-17T23:41:01.497887+0000","last_peered":"2024-09-17T23:41:01.497887+0000","last_clean":"2024-09-17T23:41:01.497887+0000","last_became_active":"2024-09-17T23:05:13.615287+0000","last_became_peered":"2024-09-17T23:05:13.615287+0000","last_unstale":"2024-09-17T23:41:01.497887+0000","last_undegraded":"2024-09-17T23:41:01.497887+0000","last_fullsized":"2024-09-17T23:41:01.497887+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:41:01.497811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:41:01.497811+0000","last_clean_scrub_stamp":"2024-09-17T23:41:01.497811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T07:23:48.834941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031014299999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1029,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-17T23:41:02.516188+0000","last_change":"2024-09-17T23:41:02.516228+0000","last_active":"2024-09-17T23:41:02.516188+0000","last_peered":"2024-09-17T23:41:02.516188+0000","last_clean":"2024-09-17T23:41:02.516188+0000","last_became_active":"2024-09-17T23:05:13.615033+0000","last_became_peered":"2024-09-17T23:05:13.615033+0000","last_unstale":"2024-09-17T23:41:02.516188+0000","last_undegraded":"2024-09-17T23:41:02.516188+0000","last_fullsized":"2024-09-17T23:41:02.516188+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:41:02.516049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:41:02.516049+0000","last_clean_scrub_stamp":"2024-09-17T23:41:02.516049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T05:41:48.507380+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00031522499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"509'2","reported_seq":1031,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-17T23:41:01.925859+0000","last_change":"2024-09-17T23:41:01.925934+0000","last_active":"2024-09-17T23:41:01.925859+0000","last_peered":"2024-09-17T23:41:01.925859+0000","last_clean":"2024-09-17T23:41:01.925859+0000","last_became_active":"2024-09-17T23:05:13.609294+0000","last_became_peered":"2024-09-17T23:05:13.609294+0000","last_unstale":"2024-09-17T23:41:01.925859+0000","last_undegraded":"2024-09-17T23:41:01.925859+0000","last_fullsized":"2024-09-17T23:41:01.925859+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"509'2","last_scrub_stamp":"2024-09-17T23:41:01.925757+0000","last_deep_scrub":"509'2","last_deep_scrub_stamp":"2024-09-17T23:41:01.925757+0000","last_clean_scrub_stamp":"2024-09-17T23:41:01.925757+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T00:35:06.716527+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00092743800000000005,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"509'99","reported_seq":1159,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-17T23:41:00.949540+0000","last_change":"2024-09-17T23:41:00.949570+0000","last_active":"2024-09-17T23:41:00.949540+0000","last_peered":"2024-09-17T23:41:00.949540+0000","last_clean":"2024-09-17T23:41:00.949540+0000","last_became_active":"2024-09-17T23:05:09.701419+0000","last_became_peered":"2024-09-17T23:05:09.701419+0000","last_unstale":"2024-09-17T23:41:00.949540+0000","last_undegraded":"2024-09-17T23:41:00.949540+0000","last_fullsized":"2024-09-17T23:41:00.949540+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"509'99","last_scrub_stamp":"2024-09-17T23:41:00.949473+0000","last_deep_scrub":"509'99","last_deep_scrub_stamp":"2024-09-17T23:41:00.949473+0000","last_clean_scrub_stamp":"2024-09-17T23:41:00.949473+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T10:31:45.546784+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":5998,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654706104,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1449620,"kb_used_data":5892,"kb_used_omap":1,"kb_used_meta":1443710,"kb_avail":92922220,"statfs":{"total":96636764160,"available":95152353280,"internally_reserved":0,"allocated":6033408,"data_stored":3009050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1478359493},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706103,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1400336,"kb_used_data":5952,"kb_used_omap":1,"kb_used_meta":1394366,"kb_avail":92971504,"statfs":{"total":96636764160,"available":95202820096,"internally_reserved":0,"allocated":6094848,"data_stored":3599437,"data_compressed":2999,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":1427831237},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706102,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1685264,"kb_used_data":5952,"kb_used_omap":1,"kb_used_meta":1679294,"kb_avail":92686576,"statfs":{"total":96636764160,"available":94911053824,"internally_reserved":0,"allocated":6094848,"data_stored":3599437,"data_compressed":2999,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":1719597509},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2999,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2999,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T23:41:21.797 DEBUG:teuthology.orchestra.run.smithi038:> sudo ceph --cluster cluster2 config set global mon_health_to_clog false 2024-09-17T23:41:22.103 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-17T23:41:22.104 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-17T23:41:22.104 DEBUG:tasks.ceph.cluster2.osd.0:waiting for process to exit 2024-09-17T23:41:22.104 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T23:41:22.179 INFO:tasks.ceph.cluster2.osd.0:Stopped 2024-09-17T23:41:22.179 DEBUG:tasks.ceph.cluster2.osd.1:waiting for process to exit 2024-09-17T23:41:22.179 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T23:41:22.258 INFO:tasks.ceph.cluster2.osd.1:Stopped 2024-09-17T23:41:22.258 DEBUG:tasks.ceph.cluster2.osd.2:waiting for process to exit 2024-09-17T23:41:22.258 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T23:41:22.344 INFO:tasks.ceph.cluster2.osd.2:Stopped 2024-09-17T23:41:22.344 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-17T23:41:22.344 DEBUG:tasks.ceph.cluster2.mgr.x:waiting for process to exit 2024-09-17T23:41:22.344 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T23:41:22.374 INFO:tasks.ceph.cluster2.mgr.x:Stopped 2024-09-17T23:41:22.421 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-17T23:41:22.422 DEBUG:tasks.ceph.cluster2.mon.a:waiting for process to exit 2024-09-17T23:41:22.422 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T23:41:22.434 INFO:tasks.ceph.cluster2.mon.a:Stopped 2024-09-17T23:41:22.435 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-17T23:41:22.435 DEBUG:teuthology.orchestra.run.smithi038:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/cluster2.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2024-09-17T23:41:22.541 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-0 on ubuntu@smithi038.front.sepia.ceph.com 2024-09-17T23:41:22.542 DEBUG:teuthology.orchestra.run.smithi038:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-0 2024-09-17T23:41:22.779 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-1 on ubuntu@smithi038.front.sepia.ceph.com 2024-09-17T23:41:22.779 DEBUG:teuthology.orchestra.run.smithi038:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-1 2024-09-17T23:41:22.931 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-2 on ubuntu@smithi038.front.sepia.ceph.com 2024-09-17T23:41:22.931 DEBUG:teuthology.orchestra.run.smithi038:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-2 2024-09-17T23:41:23.062 INFO:tasks.ceph:Archiving mon data... 2024-09-17T23:41:23.063 DEBUG:teuthology.misc:Transferring archived files from smithi038:/var/lib/ceph/mon/cluster2-a to /home/teuthworker/archive/teuthology-2024-09-17_21:16:02-rbd-squid-distro-default-smithi/7909572/data/cluster2.mon.a.tgz 2024-09-17T23:41:23.063 DEBUG:teuthology.orchestra.run.smithi038:> mktemp 2024-09-17T23:41:23.080 INFO:teuthology.orchestra.run.smithi038.stdout:/tmp/tmp.bcrmoCgyR2 2024-09-17T23:41:23.081 DEBUG:teuthology.orchestra.run.smithi038:> sudo tar cz -f - -C /var/lib/ceph/mon/cluster2-a -- . > /tmp/tmp.bcrmoCgyR2 2024-09-17T23:41:23.251 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod 0666 /tmp/tmp.bcrmoCgyR2 2024-09-17T23:41:23.368 DEBUG:teuthology.orchestra.remote:smithi038:/tmp/tmp.bcrmoCgyR2 is 437KB 2024-09-17T23:41:23.465 DEBUG:teuthology.orchestra.run.smithi038:> rm -fr /tmp/tmp.bcrmoCgyR2 2024-09-17T23:41:23.480 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-17T23:41:23.480 DEBUG:teuthology.orchestra.run.smithi006:> sudo rm -rf -- /etc/ceph/cluster2.conf /etc/ceph/cluster2.keyring /home/ubuntu/cephtest/cluster2.data /home/ubuntu/cephtest/cluster2.monmap /home/ubuntu/cephtest/../*.pid 2024-09-17T23:41:23.483 DEBUG:teuthology.orchestra.run.smithi038:> sudo rm -rf -- /etc/ceph/cluster2.conf /etc/ceph/cluster2.keyring /home/ubuntu/cephtest/cluster2.data /home/ubuntu/cephtest/cluster2.monmap /home/ubuntu/cephtest/../*.pid 2024-09-17T23:41:23.545 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-17T23:41:23.567 INFO:tasks.ceph.ceph_manager.cluster1:waiting for clean 2024-09-17T23:41:23.567 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-17T23:41:23.868 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:41:23.868 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-09-17T23:41:23.880 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":1572,"stamp":"2024-09-17T23:41:22.091675+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":451864,"kb_used_data":11352,"kb_used_omap":4,"kb_used_meta":440443,"kb_avail":282663656,"statfs":{"total":289910292480,"available":289447583744,"internally_reserved":0,"allocated":11624448,"data_stored":7328818,"data_compressed":5796,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4785,"internal_metadata":451013967},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002312"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":934,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-09-17T23:40:55.443433+0000","last_change":"2024-09-17T23:04:33.715156+0000","last_active":"2024-09-17T23:40:55.443433+0000","last_peered":"2024-09-17T23:40:55.443433+0000","last_clean":"2024-09-17T23:40:55.443433+0000","last_became_active":"2024-09-17T23:04:31.700751+0000","last_became_peered":"2024-09-17T23:04:31.700751+0000","last_unstale":"2024-09-17T23:40:55.443433+0000","last_undegraded":"2024-09-17T23:40:55.443433+0000","last_fullsized":"2024-09-17T23:40:55.443433+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:04:30.688961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:04:30.688961+0000","last_clean_scrub_stamp":"2024-09-17T23:04:30.688961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T00:01:28.967464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00096591400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":934,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-09-17T23:40:55.447293+0000","last_change":"2024-09-17T23:04:33.714782+0000","last_active":"2024-09-17T23:40:55.447293+0000","last_peered":"2024-09-17T23:40:55.447293+0000","last_clean":"2024-09-17T23:40:55.447293+0000","last_became_active":"2024-09-17T23:04:31.701763+0000","last_became_peered":"2024-09-17T23:04:31.701763+0000","last_unstale":"2024-09-17T23:40:55.447293+0000","last_undegraded":"2024-09-17T23:40:55.447293+0000","last_fullsized":"2024-09-17T23:40:55.447293+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:04:30.688961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:04:30.688961+0000","last_clean_scrub_stamp":"2024-09-17T23:04:30.688961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T05:25:40.855629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065844999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":935,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-09-17T23:40:55.447113+0000","last_change":"2024-09-17T23:04:33.715201+0000","last_active":"2024-09-17T23:40:55.447113+0000","last_peered":"2024-09-17T23:40:55.447113+0000","last_clean":"2024-09-17T23:40:55.447113+0000","last_became_active":"2024-09-17T23:04:31.703638+0000","last_became_peered":"2024-09-17T23:04:31.703638+0000","last_unstale":"2024-09-17T23:40:55.447113+0000","last_undegraded":"2024-09-17T23:40:55.447113+0000","last_fullsized":"2024-09-17T23:40:55.447113+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:04:30.688961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:04:30.688961+0000","last_clean_scrub_stamp":"2024-09-17T23:04:30.688961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T06:57:29.183690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097086099999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":935,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-09-17T23:40:55.443266+0000","last_change":"2024-09-17T23:04:33.714901+0000","last_active":"2024-09-17T23:40:55.443266+0000","last_peered":"2024-09-17T23:40:55.443266+0000","last_clean":"2024-09-17T23:40:55.443266+0000","last_became_active":"2024-09-17T23:04:31.702321+0000","last_became_peered":"2024-09-17T23:04:31.702321+0000","last_unstale":"2024-09-17T23:40:55.443266+0000","last_undegraded":"2024-09-17T23:40:55.443266+0000","last_fullsized":"2024-09-17T23:40:55.443266+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:04:30.688961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:04:30.688961+0000","last_clean_scrub_stamp":"2024-09-17T23:04:30.688961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T09:51:38.352425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00072955299999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":938,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-09-17T23:40:56.695667+0000","last_change":"2024-09-17T23:04:33.710732+0000","last_active":"2024-09-17T23:40:56.695667+0000","last_peered":"2024-09-17T23:40:56.695667+0000","last_clean":"2024-09-17T23:40:56.695667+0000","last_became_active":"2024-09-17T23:04:31.701189+0000","last_became_peered":"2024-09-17T23:04:31.701189+0000","last_unstale":"2024-09-17T23:40:56.695667+0000","last_undegraded":"2024-09-17T23:40:56.695667+0000","last_fullsized":"2024-09-17T23:40:56.695667+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:04:30.688961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:04:30.688961+0000","last_clean_scrub_stamp":"2024-09-17T23:04:30.688961+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T02:38:23.106589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010931420000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":936,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-09-17T23:40:56.526390+0000","last_change":"2024-09-17T23:04:33.939593+0000","last_active":"2024-09-17T23:40:56.526390+0000","last_peered":"2024-09-17T23:40:56.526390+0000","last_clean":"2024-09-17T23:40:56.526390+0000","last_became_active":"2024-09-17T23:04:31.701197+0000","last_became_peered":"2024-09-17T23:04:31.701197+0000","last_unstale":"2024-09-17T23:40:56.526390+0000","last_undegraded":"2024-09-17T23:40:56.526390+0000","last_fullsized":"2024-09-17T23:40:56.526390+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:04:30.688961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:04:30.688961+0000","last_clean_scrub_stamp":"2024-09-17T23:04:30.688961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T08:19:18.099975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032506599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":937,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-09-17T23:40:56.526390+0000","last_change":"2024-09-17T23:04:33.939656+0000","last_active":"2024-09-17T23:40:56.526390+0000","last_peered":"2024-09-17T23:40:56.526390+0000","last_clean":"2024-09-17T23:40:56.526390+0000","last_became_active":"2024-09-17T23:04:31.702476+0000","last_became_peered":"2024-09-17T23:04:31.702476+0000","last_unstale":"2024-09-17T23:40:56.526390+0000","last_undegraded":"2024-09-17T23:40:56.526390+0000","last_fullsized":"2024-09-17T23:40:56.526390+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:04:30.688961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:04:30.688961+0000","last_clean_scrub_stamp":"2024-09-17T23:04:30.688961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T06:57:19.469661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038066100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":936,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-09-17T23:40:55.444755+0000","last_change":"2024-09-17T23:04:33.714583+0000","last_active":"2024-09-17T23:40:55.444755+0000","last_peered":"2024-09-17T23:40:55.444755+0000","last_clean":"2024-09-17T23:40:55.444755+0000","last_became_active":"2024-09-17T23:04:31.700794+0000","last_became_peered":"2024-09-17T23:04:31.700794+0000","last_unstale":"2024-09-17T23:40:55.444755+0000","last_undegraded":"2024-09-17T23:40:55.444755+0000","last_fullsized":"2024-09-17T23:40:55.444755+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:04:30.688961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:04:30.688961+0000","last_clean_scrub_stamp":"2024-09-17T23:04:30.688961+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T09:53:38.917955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051287900000000005,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":1063,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-09-17T23:40:55.445921+0000","last_change":"2024-09-17T23:04:27.966616+0000","last_active":"2024-09-17T23:40:55.445921+0000","last_peered":"2024-09-17T23:40:55.445921+0000","last_clean":"2024-09-17T23:40:55.445921+0000","last_became_active":"2024-09-17T23:04:27.965566+0000","last_became_peered":"2024-09-17T23:04:27.965566+0000","last_unstale":"2024-09-17T23:40:55.445921+0000","last_undegraded":"2024-09-17T23:40:55.445921+0000","last_fullsized":"2024-09-17T23:40:55.445921+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:04:26.677035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:04:26.677035+0000","last_clean_scrub_stamp":"2024-09-17T23:04:26.677035+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T10:58:45.080759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":5796,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738815,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":151480,"kb_used_data":3744,"kb_used_omap":1,"kb_used_meta":147710,"kb_avail":94220360,"statfs":{"total":96636764160,"available":96481648640,"internally_reserved":0,"allocated":3833856,"data_stored":2049348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":151255493},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738817,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":143216,"kb_used_data":3804,"kb_used_omap":1,"kb_used_meta":139390,"kb_avail":94228624,"statfs":{"total":96636764160,"available":96490110976,"internally_reserved":0,"allocated":3895296,"data_stored":2639735,"data_compressed":2898,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":142735813},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738815,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":157168,"kb_used_data":3804,"kb_used_omap":1,"kb_used_meta":153342,"kb_avail":94214672,"statfs":{"total":96636764160,"available":96475824128,"internally_reserved":0,"allocated":3895296,"data_stored":2639735,"data_compressed":2898,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":157022661},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2898,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2898,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T23:41:23.881 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-17T23:41:24.169 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:41:24.169 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-09-17T23:41:24.181 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":1573,"stamp":"2024-09-17T23:41:24.091998+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":451864,"kb_used_data":11352,"kb_used_omap":4,"kb_used_meta":440443,"kb_avail":282663656,"statfs":{"total":289910292480,"available":289447583744,"internally_reserved":0,"allocated":11624448,"data_stored":7328818,"data_compressed":5796,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4785,"internal_metadata":451013967},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002254"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":934,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-09-17T23:40:55.443433+0000","last_change":"2024-09-17T23:04:33.715156+0000","last_active":"2024-09-17T23:40:55.443433+0000","last_peered":"2024-09-17T23:40:55.443433+0000","last_clean":"2024-09-17T23:40:55.443433+0000","last_became_active":"2024-09-17T23:04:31.700751+0000","last_became_peered":"2024-09-17T23:04:31.700751+0000","last_unstale":"2024-09-17T23:40:55.443433+0000","last_undegraded":"2024-09-17T23:40:55.443433+0000","last_fullsized":"2024-09-17T23:40:55.443433+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:04:30.688961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:04:30.688961+0000","last_clean_scrub_stamp":"2024-09-17T23:04:30.688961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T00:01:28.967464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00096591400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":934,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-09-17T23:40:55.447293+0000","last_change":"2024-09-17T23:04:33.714782+0000","last_active":"2024-09-17T23:40:55.447293+0000","last_peered":"2024-09-17T23:40:55.447293+0000","last_clean":"2024-09-17T23:40:55.447293+0000","last_became_active":"2024-09-17T23:04:31.701763+0000","last_became_peered":"2024-09-17T23:04:31.701763+0000","last_unstale":"2024-09-17T23:40:55.447293+0000","last_undegraded":"2024-09-17T23:40:55.447293+0000","last_fullsized":"2024-09-17T23:40:55.447293+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:04:30.688961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:04:30.688961+0000","last_clean_scrub_stamp":"2024-09-17T23:04:30.688961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T05:25:40.855629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065844999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":935,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-09-17T23:40:55.447113+0000","last_change":"2024-09-17T23:04:33.715201+0000","last_active":"2024-09-17T23:40:55.447113+0000","last_peered":"2024-09-17T23:40:55.447113+0000","last_clean":"2024-09-17T23:40:55.447113+0000","last_became_active":"2024-09-17T23:04:31.703638+0000","last_became_peered":"2024-09-17T23:04:31.703638+0000","last_unstale":"2024-09-17T23:40:55.447113+0000","last_undegraded":"2024-09-17T23:40:55.447113+0000","last_fullsized":"2024-09-17T23:40:55.447113+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:04:30.688961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:04:30.688961+0000","last_clean_scrub_stamp":"2024-09-17T23:04:30.688961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T06:57:29.183690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097086099999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":935,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-09-17T23:40:55.443266+0000","last_change":"2024-09-17T23:04:33.714901+0000","last_active":"2024-09-17T23:40:55.443266+0000","last_peered":"2024-09-17T23:40:55.443266+0000","last_clean":"2024-09-17T23:40:55.443266+0000","last_became_active":"2024-09-17T23:04:31.702321+0000","last_became_peered":"2024-09-17T23:04:31.702321+0000","last_unstale":"2024-09-17T23:40:55.443266+0000","last_undegraded":"2024-09-17T23:40:55.443266+0000","last_fullsized":"2024-09-17T23:40:55.443266+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:04:30.688961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:04:30.688961+0000","last_clean_scrub_stamp":"2024-09-17T23:04:30.688961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T09:51:38.352425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00072955299999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":938,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-09-17T23:40:56.695667+0000","last_change":"2024-09-17T23:04:33.710732+0000","last_active":"2024-09-17T23:40:56.695667+0000","last_peered":"2024-09-17T23:40:56.695667+0000","last_clean":"2024-09-17T23:40:56.695667+0000","last_became_active":"2024-09-17T23:04:31.701189+0000","last_became_peered":"2024-09-17T23:04:31.701189+0000","last_unstale":"2024-09-17T23:40:56.695667+0000","last_undegraded":"2024-09-17T23:40:56.695667+0000","last_fullsized":"2024-09-17T23:40:56.695667+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:04:30.688961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:04:30.688961+0000","last_clean_scrub_stamp":"2024-09-17T23:04:30.688961+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T02:38:23.106589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010931420000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":936,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-09-17T23:40:56.526390+0000","last_change":"2024-09-17T23:04:33.939593+0000","last_active":"2024-09-17T23:40:56.526390+0000","last_peered":"2024-09-17T23:40:56.526390+0000","last_clean":"2024-09-17T23:40:56.526390+0000","last_became_active":"2024-09-17T23:04:31.701197+0000","last_became_peered":"2024-09-17T23:04:31.701197+0000","last_unstale":"2024-09-17T23:40:56.526390+0000","last_undegraded":"2024-09-17T23:40:56.526390+0000","last_fullsized":"2024-09-17T23:40:56.526390+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:04:30.688961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:04:30.688961+0000","last_clean_scrub_stamp":"2024-09-17T23:04:30.688961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T08:19:18.099975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032506599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":937,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-09-17T23:40:56.526390+0000","last_change":"2024-09-17T23:04:33.939656+0000","last_active":"2024-09-17T23:40:56.526390+0000","last_peered":"2024-09-17T23:40:56.526390+0000","last_clean":"2024-09-17T23:40:56.526390+0000","last_became_active":"2024-09-17T23:04:31.702476+0000","last_became_peered":"2024-09-17T23:04:31.702476+0000","last_unstale":"2024-09-17T23:40:56.526390+0000","last_undegraded":"2024-09-17T23:40:56.526390+0000","last_fullsized":"2024-09-17T23:40:56.526390+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:04:30.688961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:04:30.688961+0000","last_clean_scrub_stamp":"2024-09-17T23:04:30.688961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T06:57:19.469661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038066100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":936,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-09-17T23:40:55.444755+0000","last_change":"2024-09-17T23:04:33.714583+0000","last_active":"2024-09-17T23:40:55.444755+0000","last_peered":"2024-09-17T23:40:55.444755+0000","last_clean":"2024-09-17T23:40:55.444755+0000","last_became_active":"2024-09-17T23:04:31.700794+0000","last_became_peered":"2024-09-17T23:04:31.700794+0000","last_unstale":"2024-09-17T23:40:55.444755+0000","last_undegraded":"2024-09-17T23:40:55.444755+0000","last_fullsized":"2024-09-17T23:40:55.444755+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:04:30.688961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:04:30.688961+0000","last_clean_scrub_stamp":"2024-09-17T23:04:30.688961+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T09:53:38.917955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051287900000000005,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":1063,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-09-17T23:40:55.445921+0000","last_change":"2024-09-17T23:04:27.966616+0000","last_active":"2024-09-17T23:40:55.445921+0000","last_peered":"2024-09-17T23:40:55.445921+0000","last_clean":"2024-09-17T23:40:55.445921+0000","last_became_active":"2024-09-17T23:04:27.965566+0000","last_became_peered":"2024-09-17T23:04:27.965566+0000","last_unstale":"2024-09-17T23:40:55.445921+0000","last_undegraded":"2024-09-17T23:40:55.445921+0000","last_fullsized":"2024-09-17T23:40:55.445921+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:04:26.677035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:04:26.677035+0000","last_clean_scrub_stamp":"2024-09-17T23:04:26.677035+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T10:58:45.080759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":5796,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738815,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":151480,"kb_used_data":3744,"kb_used_omap":1,"kb_used_meta":147710,"kb_avail":94220360,"statfs":{"total":96636764160,"available":96481648640,"internally_reserved":0,"allocated":3833856,"data_stored":2049348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":151255493},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738817,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":143216,"kb_used_data":3804,"kb_used_omap":1,"kb_used_meta":139390,"kb_avail":94228624,"statfs":{"total":96636764160,"available":96490110976,"internally_reserved":0,"allocated":3895296,"data_stored":2639735,"data_compressed":2898,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":142735813},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738815,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":157168,"kb_used_data":3804,"kb_used_omap":1,"kb_used_meta":153342,"kb_avail":94214672,"statfs":{"total":96636764160,"available":96475824128,"internally_reserved":0,"allocated":3895296,"data_stored":2639735,"data_compressed":2898,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":157022661},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2898,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2898,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T23:41:24.182 INFO:tasks.ceph.ceph_manager.cluster1:clean! 2024-09-17T23:41:24.182 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-17T23:41:24.472 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:41:24.472 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-09-17T23:41:24.484 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":1573,"stamp":"2024-09-17T23:41:24.091998+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":451864,"kb_used_data":11352,"kb_used_omap":4,"kb_used_meta":440443,"kb_avail":282663656,"statfs":{"total":289910292480,"available":289447583744,"internally_reserved":0,"allocated":11624448,"data_stored":7328818,"data_compressed":5796,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4785,"internal_metadata":451013967},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002254"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":934,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-09-17T23:40:55.443433+0000","last_change":"2024-09-17T23:04:33.715156+0000","last_active":"2024-09-17T23:40:55.443433+0000","last_peered":"2024-09-17T23:40:55.443433+0000","last_clean":"2024-09-17T23:40:55.443433+0000","last_became_active":"2024-09-17T23:04:31.700751+0000","last_became_peered":"2024-09-17T23:04:31.700751+0000","last_unstale":"2024-09-17T23:40:55.443433+0000","last_undegraded":"2024-09-17T23:40:55.443433+0000","last_fullsized":"2024-09-17T23:40:55.443433+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:04:30.688961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:04:30.688961+0000","last_clean_scrub_stamp":"2024-09-17T23:04:30.688961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T00:01:28.967464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00096591400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":934,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-09-17T23:40:55.447293+0000","last_change":"2024-09-17T23:04:33.714782+0000","last_active":"2024-09-17T23:40:55.447293+0000","last_peered":"2024-09-17T23:40:55.447293+0000","last_clean":"2024-09-17T23:40:55.447293+0000","last_became_active":"2024-09-17T23:04:31.701763+0000","last_became_peered":"2024-09-17T23:04:31.701763+0000","last_unstale":"2024-09-17T23:40:55.447293+0000","last_undegraded":"2024-09-17T23:40:55.447293+0000","last_fullsized":"2024-09-17T23:40:55.447293+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:04:30.688961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:04:30.688961+0000","last_clean_scrub_stamp":"2024-09-17T23:04:30.688961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T05:25:40.855629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065844999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":935,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-09-17T23:40:55.447113+0000","last_change":"2024-09-17T23:04:33.715201+0000","last_active":"2024-09-17T23:40:55.447113+0000","last_peered":"2024-09-17T23:40:55.447113+0000","last_clean":"2024-09-17T23:40:55.447113+0000","last_became_active":"2024-09-17T23:04:31.703638+0000","last_became_peered":"2024-09-17T23:04:31.703638+0000","last_unstale":"2024-09-17T23:40:55.447113+0000","last_undegraded":"2024-09-17T23:40:55.447113+0000","last_fullsized":"2024-09-17T23:40:55.447113+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:04:30.688961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:04:30.688961+0000","last_clean_scrub_stamp":"2024-09-17T23:04:30.688961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T06:57:29.183690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097086099999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":935,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-09-17T23:40:55.443266+0000","last_change":"2024-09-17T23:04:33.714901+0000","last_active":"2024-09-17T23:40:55.443266+0000","last_peered":"2024-09-17T23:40:55.443266+0000","last_clean":"2024-09-17T23:40:55.443266+0000","last_became_active":"2024-09-17T23:04:31.702321+0000","last_became_peered":"2024-09-17T23:04:31.702321+0000","last_unstale":"2024-09-17T23:40:55.443266+0000","last_undegraded":"2024-09-17T23:40:55.443266+0000","last_fullsized":"2024-09-17T23:40:55.443266+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:04:30.688961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:04:30.688961+0000","last_clean_scrub_stamp":"2024-09-17T23:04:30.688961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T09:51:38.352425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00072955299999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":938,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-09-17T23:40:56.695667+0000","last_change":"2024-09-17T23:04:33.710732+0000","last_active":"2024-09-17T23:40:56.695667+0000","last_peered":"2024-09-17T23:40:56.695667+0000","last_clean":"2024-09-17T23:40:56.695667+0000","last_became_active":"2024-09-17T23:04:31.701189+0000","last_became_peered":"2024-09-17T23:04:31.701189+0000","last_unstale":"2024-09-17T23:40:56.695667+0000","last_undegraded":"2024-09-17T23:40:56.695667+0000","last_fullsized":"2024-09-17T23:40:56.695667+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:04:30.688961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:04:30.688961+0000","last_clean_scrub_stamp":"2024-09-17T23:04:30.688961+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T02:38:23.106589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010931420000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":936,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-09-17T23:40:56.526390+0000","last_change":"2024-09-17T23:04:33.939593+0000","last_active":"2024-09-17T23:40:56.526390+0000","last_peered":"2024-09-17T23:40:56.526390+0000","last_clean":"2024-09-17T23:40:56.526390+0000","last_became_active":"2024-09-17T23:04:31.701197+0000","last_became_peered":"2024-09-17T23:04:31.701197+0000","last_unstale":"2024-09-17T23:40:56.526390+0000","last_undegraded":"2024-09-17T23:40:56.526390+0000","last_fullsized":"2024-09-17T23:40:56.526390+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:04:30.688961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:04:30.688961+0000","last_clean_scrub_stamp":"2024-09-17T23:04:30.688961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T08:19:18.099975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032506599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":937,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-09-17T23:40:56.526390+0000","last_change":"2024-09-17T23:04:33.939656+0000","last_active":"2024-09-17T23:40:56.526390+0000","last_peered":"2024-09-17T23:40:56.526390+0000","last_clean":"2024-09-17T23:40:56.526390+0000","last_became_active":"2024-09-17T23:04:31.702476+0000","last_became_peered":"2024-09-17T23:04:31.702476+0000","last_unstale":"2024-09-17T23:40:56.526390+0000","last_undegraded":"2024-09-17T23:40:56.526390+0000","last_fullsized":"2024-09-17T23:40:56.526390+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:04:30.688961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:04:30.688961+0000","last_clean_scrub_stamp":"2024-09-17T23:04:30.688961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T06:57:19.469661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038066100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":936,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-09-17T23:40:55.444755+0000","last_change":"2024-09-17T23:04:33.714583+0000","last_active":"2024-09-17T23:40:55.444755+0000","last_peered":"2024-09-17T23:40:55.444755+0000","last_clean":"2024-09-17T23:40:55.444755+0000","last_became_active":"2024-09-17T23:04:31.700794+0000","last_became_peered":"2024-09-17T23:04:31.700794+0000","last_unstale":"2024-09-17T23:40:55.444755+0000","last_undegraded":"2024-09-17T23:40:55.444755+0000","last_fullsized":"2024-09-17T23:40:55.444755+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:04:30.688961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:04:30.688961+0000","last_clean_scrub_stamp":"2024-09-17T23:04:30.688961+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T09:53:38.917955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051287900000000005,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":1063,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-09-17T23:40:55.445921+0000","last_change":"2024-09-17T23:04:27.966616+0000","last_active":"2024-09-17T23:40:55.445921+0000","last_peered":"2024-09-17T23:40:55.445921+0000","last_clean":"2024-09-17T23:40:55.445921+0000","last_became_active":"2024-09-17T23:04:27.965566+0000","last_became_peered":"2024-09-17T23:04:27.965566+0000","last_unstale":"2024-09-17T23:40:55.445921+0000","last_undegraded":"2024-09-17T23:40:55.445921+0000","last_fullsized":"2024-09-17T23:40:55.445921+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:04:26.677035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:04:26.677035+0000","last_clean_scrub_stamp":"2024-09-17T23:04:26.677035+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T10:58:45.080759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":5796,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738815,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":151480,"kb_used_data":3744,"kb_used_omap":1,"kb_used_meta":147710,"kb_avail":94220360,"statfs":{"total":96636764160,"available":96481648640,"internally_reserved":0,"allocated":3833856,"data_stored":2049348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":151255493},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738817,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":143216,"kb_used_data":3804,"kb_used_omap":1,"kb_used_meta":139390,"kb_avail":94228624,"statfs":{"total":96636764160,"available":96490110976,"internally_reserved":0,"allocated":3895296,"data_stored":2639735,"data_compressed":2898,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":142735813},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738815,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":157168,"kb_used_data":3804,"kb_used_omap":1,"kb_used_meta":153342,"kb_avail":94214672,"statfs":{"total":96636764160,"available":96475824128,"internally_reserved":0,"allocated":3895296,"data_stored":2639735,"data_compressed":2898,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":157022661},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2898,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2898,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T23:41:24.485 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-09-17T23:41:24.775 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:41:24.775 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":466,"fsid":"545dc37d-b29c-4f04-aad4-46458bcf9f65","created":"2024-09-17T23:04:15.024128+0000","modified":"2024-09-17T23:40:55.433547+0000","last_up_change":"2024-09-17T23:04:24.674100+0000","last_in_change":"2024-09-17T23:04:15.924722+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T23:04:25.756881+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-17T23:04:30.549319+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"530ec6e3-0be9-4615-9d5a-2fa97fce2539","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":4294176576},{"type":"v1","addr":"172.21.15.6:6809","nonce":4294176576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":4294176576},{"type":"v1","addr":"172.21.15.6:6811","nonce":4294176576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":4294176576},{"type":"v1","addr":"172.21.15.6:6815","nonce":4294176576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":4294176576},{"type":"v1","addr":"172.21.15.6:6813","nonce":4294176576}]},"public_addr":"172.21.15.6:6809/4294176576","cluster_addr":"172.21.15.6:6811/4294176576","heartbeat_back_addr":"172.21.15.6:6815/4294176576","heartbeat_front_addr":"172.21.15.6:6813/4294176576","state":["exists","up"]},{"osd":1,"uuid":"1d1e25d4-5837-450d-aae1-a44b3233c9ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1789421019},{"type":"v1","addr":"172.21.15.6:6801","nonce":1789421019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1789421019},{"type":"v1","addr":"172.21.15.6:6803","nonce":1789421019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1789421019},{"type":"v1","addr":"172.21.15.6:6807","nonce":1789421019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1789421019},{"type":"v1","addr":"172.21.15.6:6805","nonce":1789421019}]},"public_addr":"172.21.15.6:6801/1789421019","cluster_addr":"172.21.15.6:6803/1789421019","heartbeat_back_addr":"172.21.15.6:6807/1789421019","heartbeat_front_addr":"172.21.15.6:6805/1789421019","state":["exists","up"]},{"osd":2,"uuid":"9540a0fe-29b1-4de7-b617-a765d3f99fc5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":2756875437},{"type":"v1","addr":"172.21.15.6:6817","nonce":2756875437}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":2756875437},{"type":"v1","addr":"172.21.15.6:6819","nonce":2756875437}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":2756875437},{"type":"v1","addr":"172.21.15.6:6823","nonce":2756875437}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":2756875437},{"type":"v1","addr":"172.21.15.6:6821","nonce":2756875437}]},"public_addr":"172.21.15.6:6817/2756875437","cluster_addr":"172.21.15.6:6819/2756875437","heartbeat_back_addr":"172.21.15.6:6823/2756875437","heartbeat_front_addr":"172.21.15.6:6821/2756875437","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T23:04:22.254942+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T23:04:21.768119+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T23:04:21.962217+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":194,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T23:41:25.789 INFO:tasks.ceph:Scrubbing cluster1.osd.0 2024-09-17T23:41:25.789 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-17T23:41:25.929 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-17T23:41:25.929 INFO:teuthology.orchestra.run.smithi006.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-17T23:41:25.929 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-17T23:41:25.940 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 0 2024-09-17T23:41:26.233 INFO:teuthology.orchestra.run.smithi006.stderr:instructed osd(s) 0 to deep-scrub 2024-09-17T23:41:26.246 INFO:tasks.ceph:Scrubbing cluster1.osd.1 2024-09-17T23:41:26.246 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-17T23:41:26.372 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-17T23:41:26.372 INFO:teuthology.orchestra.run.smithi006.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-17T23:41:26.372 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-17T23:41:26.383 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 1 2024-09-17T23:41:26.676 INFO:teuthology.orchestra.run.smithi006.stderr:instructed osd(s) 1 to deep-scrub 2024-09-17T23:41:26.689 INFO:tasks.ceph:Scrubbing cluster1.osd.2 2024-09-17T23:41:26.690 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-17T23:41:26.816 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-17T23:41:26.816 INFO:teuthology.orchestra.run.smithi006.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-17T23:41:26.816 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-17T23:41:26.827 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 2 2024-09-17T23:41:27.116 INFO:teuthology.orchestra.run.smithi006.stderr:instructed osd(s) 2 to deep-scrub 2024-09-17T23:41:27.130 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-17T23:41:27.418 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:41:27.418 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-09-17T23:41:27.431 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":1574,"stamp":"2024-09-17T23:41:26.092232+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":451864,"kb_used_data":11352,"kb_used_omap":4,"kb_used_meta":440443,"kb_avail":282663656,"statfs":{"total":289910292480,"available":289447583744,"internally_reserved":0,"allocated":11624448,"data_stored":7328818,"data_compressed":5796,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4785,"internal_metadata":451013967},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002076"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":935,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-09-17T23:40:55.443483+0000","last_change":"2024-09-17T23:04:33.715156+0000","last_active":"2024-09-17T23:40:55.443483+0000","last_peered":"2024-09-17T23:40:55.443483+0000","last_clean":"2024-09-17T23:40:55.443483+0000","last_became_active":"2024-09-17T23:04:31.700751+0000","last_became_peered":"2024-09-17T23:04:31.700751+0000","last_unstale":"2024-09-17T23:40:55.443483+0000","last_undegraded":"2024-09-17T23:40:55.443483+0000","last_fullsized":"2024-09-17T23:40:55.443483+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:04:30.688961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:04:30.688961+0000","last_clean_scrub_stamp":"2024-09-17T23:04:30.688961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T00:01:28.967464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00096591400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":935,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-09-17T23:40:55.447341+0000","last_change":"2024-09-17T23:04:33.714782+0000","last_active":"2024-09-17T23:40:55.447341+0000","last_peered":"2024-09-17T23:40:55.447341+0000","last_clean":"2024-09-17T23:40:55.447341+0000","last_became_active":"2024-09-17T23:04:31.701763+0000","last_became_peered":"2024-09-17T23:04:31.701763+0000","last_unstale":"2024-09-17T23:40:55.447341+0000","last_undegraded":"2024-09-17T23:40:55.447341+0000","last_fullsized":"2024-09-17T23:40:55.447341+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:04:30.688961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:04:30.688961+0000","last_clean_scrub_stamp":"2024-09-17T23:04:30.688961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T05:25:40.855629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065844999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":935,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-09-17T23:40:55.447113+0000","last_change":"2024-09-17T23:04:33.715201+0000","last_active":"2024-09-17T23:40:55.447113+0000","last_peered":"2024-09-17T23:40:55.447113+0000","last_clean":"2024-09-17T23:40:55.447113+0000","last_became_active":"2024-09-17T23:04:31.703638+0000","last_became_peered":"2024-09-17T23:04:31.703638+0000","last_unstale":"2024-09-17T23:40:55.447113+0000","last_undegraded":"2024-09-17T23:40:55.447113+0000","last_fullsized":"2024-09-17T23:40:55.447113+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:04:30.688961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:04:30.688961+0000","last_clean_scrub_stamp":"2024-09-17T23:04:30.688961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T06:57:29.183690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097086099999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":935,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-09-17T23:40:55.443266+0000","last_change":"2024-09-17T23:04:33.714901+0000","last_active":"2024-09-17T23:40:55.443266+0000","last_peered":"2024-09-17T23:40:55.443266+0000","last_clean":"2024-09-17T23:40:55.443266+0000","last_became_active":"2024-09-17T23:04:31.702321+0000","last_became_peered":"2024-09-17T23:04:31.702321+0000","last_unstale":"2024-09-17T23:40:55.443266+0000","last_undegraded":"2024-09-17T23:40:55.443266+0000","last_fullsized":"2024-09-17T23:40:55.443266+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:04:30.688961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:04:30.688961+0000","last_clean_scrub_stamp":"2024-09-17T23:04:30.688961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T09:51:38.352425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00072955299999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":938,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-09-17T23:40:56.695667+0000","last_change":"2024-09-17T23:04:33.710732+0000","last_active":"2024-09-17T23:40:56.695667+0000","last_peered":"2024-09-17T23:40:56.695667+0000","last_clean":"2024-09-17T23:40:56.695667+0000","last_became_active":"2024-09-17T23:04:31.701189+0000","last_became_peered":"2024-09-17T23:04:31.701189+0000","last_unstale":"2024-09-17T23:40:56.695667+0000","last_undegraded":"2024-09-17T23:40:56.695667+0000","last_fullsized":"2024-09-17T23:40:56.695667+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:04:30.688961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:04:30.688961+0000","last_clean_scrub_stamp":"2024-09-17T23:04:30.688961+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T02:38:23.106589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010931420000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":936,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-09-17T23:40:56.526390+0000","last_change":"2024-09-17T23:04:33.939593+0000","last_active":"2024-09-17T23:40:56.526390+0000","last_peered":"2024-09-17T23:40:56.526390+0000","last_clean":"2024-09-17T23:40:56.526390+0000","last_became_active":"2024-09-17T23:04:31.701197+0000","last_became_peered":"2024-09-17T23:04:31.701197+0000","last_unstale":"2024-09-17T23:40:56.526390+0000","last_undegraded":"2024-09-17T23:40:56.526390+0000","last_fullsized":"2024-09-17T23:40:56.526390+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:04:30.688961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:04:30.688961+0000","last_clean_scrub_stamp":"2024-09-17T23:04:30.688961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T08:19:18.099975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032506599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":937,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-09-17T23:40:56.526390+0000","last_change":"2024-09-17T23:04:33.939656+0000","last_active":"2024-09-17T23:40:56.526390+0000","last_peered":"2024-09-17T23:40:56.526390+0000","last_clean":"2024-09-17T23:40:56.526390+0000","last_became_active":"2024-09-17T23:04:31.702476+0000","last_became_peered":"2024-09-17T23:04:31.702476+0000","last_unstale":"2024-09-17T23:40:56.526390+0000","last_undegraded":"2024-09-17T23:40:56.526390+0000","last_fullsized":"2024-09-17T23:40:56.526390+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:04:30.688961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:04:30.688961+0000","last_clean_scrub_stamp":"2024-09-17T23:04:30.688961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T06:57:19.469661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038066100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":936,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-09-17T23:40:55.444755+0000","last_change":"2024-09-17T23:04:33.714583+0000","last_active":"2024-09-17T23:40:55.444755+0000","last_peered":"2024-09-17T23:40:55.444755+0000","last_clean":"2024-09-17T23:40:55.444755+0000","last_became_active":"2024-09-17T23:04:31.700794+0000","last_became_peered":"2024-09-17T23:04:31.700794+0000","last_unstale":"2024-09-17T23:40:55.444755+0000","last_undegraded":"2024-09-17T23:40:55.444755+0000","last_fullsized":"2024-09-17T23:40:55.444755+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:04:30.688961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:04:30.688961+0000","last_clean_scrub_stamp":"2024-09-17T23:04:30.688961+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T09:53:38.917955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051287900000000005,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":1063,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-09-17T23:40:55.445921+0000","last_change":"2024-09-17T23:04:27.966616+0000","last_active":"2024-09-17T23:40:55.445921+0000","last_peered":"2024-09-17T23:40:55.445921+0000","last_clean":"2024-09-17T23:40:55.445921+0000","last_became_active":"2024-09-17T23:04:27.965566+0000","last_became_peered":"2024-09-17T23:04:27.965566+0000","last_unstale":"2024-09-17T23:40:55.445921+0000","last_undegraded":"2024-09-17T23:40:55.445921+0000","last_fullsized":"2024-09-17T23:40:55.445921+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:04:26.677035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:04:26.677035+0000","last_clean_scrub_stamp":"2024-09-17T23:04:26.677035+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T10:58:45.080759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":5796,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738816,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":151480,"kb_used_data":3744,"kb_used_omap":1,"kb_used_meta":147710,"kb_avail":94220360,"statfs":{"total":96636764160,"available":96481648640,"internally_reserved":0,"allocated":3833856,"data_stored":2049348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":151255493},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738818,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":143216,"kb_used_data":3804,"kb_used_omap":1,"kb_used_meta":139390,"kb_avail":94228624,"statfs":{"total":96636764160,"available":96490110976,"internally_reserved":0,"allocated":3895296,"data_stored":2639735,"data_compressed":2898,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":142735813},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738816,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":157168,"kb_used_data":3804,"kb_used_omap":1,"kb_used_meta":153342,"kb_avail":94214672,"statfs":{"total":96636764160,"available":96475824128,"internally_reserved":0,"allocated":3895296,"data_stored":2639735,"data_compressed":2898,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":157022661},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2898,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2898,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T23:41:27.432 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-17T23:04:30.688961+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=23, tm_min=4, tm_sec=30, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=23, tm_min=41, tm_sec=24, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T23:41:27.432 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-09-17T23:04:30.688961+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=23, tm_min=4, tm_sec=30, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=23, tm_min=41, tm_sec=24, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T23:41:27.433 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-17T23:04:30.688961+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=23, tm_min=4, tm_sec=30, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=23, tm_min=41, tm_sec=24, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T23:41:27.433 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-09-17T23:04:30.688961+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=23, tm_min=4, tm_sec=30, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=23, tm_min=41, tm_sec=24, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T23:41:27.433 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-09-17T23:04:30.688961+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=23, tm_min=4, tm_sec=30, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=23, tm_min=41, tm_sec=24, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T23:41:27.433 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-09-17T23:04:30.688961+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=23, tm_min=4, tm_sec=30, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=23, tm_min=41, tm_sec=24, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T23:41:27.434 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-17T23:04:30.688961+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=23, tm_min=4, tm_sec=30, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=23, tm_min=41, tm_sec=24, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T23:41:27.434 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-17T23:04:30.688961+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=23, tm_min=4, tm_sec=30, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=23, tm_min=41, tm_sec=24, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T23:41:27.434 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-17T23:04:26.677035+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=23, tm_min=4, tm_sec=26, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=23, tm_min=41, tm_sec=24, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T23:41:27.434 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-17T23:41:47.436 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-17T23:41:47.739 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T23:41:47.739 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-09-17T23:41:47.752 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":1584,"stamp":"2024-09-17T23:41:46.095416+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":104,"ondisk_log_size":104,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":451848,"kb_used_data":11352,"kb_used_omap":4,"kb_used_meta":440443,"kb_avail":282663672,"statfs":{"total":289910292480,"available":289447600128,"internally_reserved":0,"allocated":11624448,"data_stored":7328818,"data_compressed":5796,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4785,"internal_metadata":451013967},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001938"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":944,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-09-17T23:41:28.205224+0000","last_change":"2024-09-17T23:41:28.205248+0000","last_active":"2024-09-17T23:41:28.205224+0000","last_peered":"2024-09-17T23:41:28.205224+0000","last_clean":"2024-09-17T23:41:28.205224+0000","last_became_active":"2024-09-17T23:04:31.700751+0000","last_became_peered":"2024-09-17T23:04:31.700751+0000","last_unstale":"2024-09-17T23:41:28.205224+0000","last_undegraded":"2024-09-17T23:41:28.205224+0000","last_fullsized":"2024-09-17T23:41:28.205224+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:41:28.205121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:41:28.205121+0000","last_clean_scrub_stamp":"2024-09-17T23:41:28.205121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T04:29:40.633764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00096591400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":944,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-09-17T23:41:29.204054+0000","last_change":"2024-09-17T23:41:29.204084+0000","last_active":"2024-09-17T23:41:29.204054+0000","last_peered":"2024-09-17T23:41:29.204054+0000","last_clean":"2024-09-17T23:41:29.204054+0000","last_became_active":"2024-09-17T23:04:31.701763+0000","last_became_peered":"2024-09-17T23:04:31.701763+0000","last_unstale":"2024-09-17T23:41:29.204054+0000","last_undegraded":"2024-09-17T23:41:29.204054+0000","last_fullsized":"2024-09-17T23:41:29.204054+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:41:29.203953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:41:29.203953+0000","last_clean_scrub_stamp":"2024-09-17T23:41:29.203953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T00:49:50.959900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065844999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":944,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-09-17T23:41:32.177219+0000","last_change":"2024-09-17T23:41:32.177241+0000","last_active":"2024-09-17T23:41:32.177219+0000","last_peered":"2024-09-17T23:41:32.177219+0000","last_clean":"2024-09-17T23:41:32.177219+0000","last_became_active":"2024-09-17T23:04:31.703638+0000","last_became_peered":"2024-09-17T23:04:31.703638+0000","last_unstale":"2024-09-17T23:41:32.177219+0000","last_undegraded":"2024-09-17T23:41:32.177219+0000","last_fullsized":"2024-09-17T23:41:32.177219+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:41:32.177126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:41:32.177126+0000","last_clean_scrub_stamp":"2024-09-17T23:41:32.177126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T02:05:29.682241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097086099999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":944,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-09-17T23:41:30.177262+0000","last_change":"2024-09-17T23:41:30.177284+0000","last_active":"2024-09-17T23:41:30.177262+0000","last_peered":"2024-09-17T23:41:30.177262+0000","last_clean":"2024-09-17T23:41:30.177262+0000","last_became_active":"2024-09-17T23:04:31.702321+0000","last_became_peered":"2024-09-17T23:04:31.702321+0000","last_unstale":"2024-09-17T23:41:30.177262+0000","last_undegraded":"2024-09-17T23:41:30.177262+0000","last_fullsized":"2024-09-17T23:41:30.177262+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:41:30.177162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:41:30.177162+0000","last_clean_scrub_stamp":"2024-09-17T23:41:30.177162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T00:27:03.854490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00072955299999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"466'3","reported_seq":948,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-09-17T23:41:26.573469+0000","last_change":"2024-09-17T23:41:26.573493+0000","last_active":"2024-09-17T23:41:26.573469+0000","last_peered":"2024-09-17T23:41:26.573469+0000","last_clean":"2024-09-17T23:41:26.573469+0000","last_became_active":"2024-09-17T23:04:31.701189+0000","last_became_peered":"2024-09-17T23:04:31.701189+0000","last_unstale":"2024-09-17T23:41:26.573469+0000","last_undegraded":"2024-09-17T23:41:26.573469+0000","last_fullsized":"2024-09-17T23:41:26.573469+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"466'3","last_scrub_stamp":"2024-09-17T23:41:26.573394+0000","last_deep_scrub":"466'3","last_deep_scrub_stamp":"2024-09-17T23:41:26.573394+0000","last_clean_scrub_stamp":"2024-09-17T23:41:26.573394+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T03:23:11.422095+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0010931420000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":945,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-09-17T23:41:27.270100+0000","last_change":"2024-09-17T23:41:27.270167+0000","last_active":"2024-09-17T23:41:27.270100+0000","last_peered":"2024-09-17T23:41:27.270100+0000","last_clean":"2024-09-17T23:41:27.270100+0000","last_became_active":"2024-09-17T23:04:31.701197+0000","last_became_peered":"2024-09-17T23:04:31.701197+0000","last_unstale":"2024-09-17T23:41:27.270100+0000","last_undegraded":"2024-09-17T23:41:27.270100+0000","last_fullsized":"2024-09-17T23:41:27.270100+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:41:27.270003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:41:27.270003+0000","last_clean_scrub_stamp":"2024-09-17T23:41:27.270003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T05:42:13.793237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032506599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":946,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-09-17T23:41:28.245980+0000","last_change":"2024-09-17T23:41:28.246041+0000","last_active":"2024-09-17T23:41:28.245980+0000","last_peered":"2024-09-17T23:41:28.245980+0000","last_clean":"2024-09-17T23:41:28.245980+0000","last_became_active":"2024-09-17T23:04:31.702476+0000","last_became_peered":"2024-09-17T23:04:31.702476+0000","last_unstale":"2024-09-17T23:41:28.245980+0000","last_undegraded":"2024-09-17T23:41:28.245980+0000","last_fullsized":"2024-09-17T23:41:28.245980+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:41:28.245874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:41:28.245874+0000","last_clean_scrub_stamp":"2024-09-17T23:41:28.245874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T01:12:11.345100+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00038066100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"466'2","reported_seq":946,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-09-17T23:41:31.160784+0000","last_change":"2024-09-17T23:41:31.160851+0000","last_active":"2024-09-17T23:41:31.160784+0000","last_peered":"2024-09-17T23:41:31.160784+0000","last_clean":"2024-09-17T23:41:31.160784+0000","last_became_active":"2024-09-17T23:04:31.700794+0000","last_became_peered":"2024-09-17T23:04:31.700794+0000","last_unstale":"2024-09-17T23:41:31.160784+0000","last_undegraded":"2024-09-17T23:41:31.160784+0000","last_fullsized":"2024-09-17T23:41:31.160784+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"466'2","last_scrub_stamp":"2024-09-17T23:41:31.160680+0000","last_deep_scrub":"466'2","last_deep_scrub_stamp":"2024-09-17T23:41:31.160680+0000","last_clean_scrub_stamp":"2024-09-17T23:41:31.160680+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T06:02:30.069694+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00051287900000000005,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"466'99","reported_seq":1074,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-09-17T23:41:27.189326+0000","last_change":"2024-09-17T23:41:27.189355+0000","last_active":"2024-09-17T23:41:27.189326+0000","last_peered":"2024-09-17T23:41:27.189326+0000","last_clean":"2024-09-17T23:41:27.189326+0000","last_became_active":"2024-09-17T23:04:27.965566+0000","last_became_peered":"2024-09-17T23:04:27.965566+0000","last_unstale":"2024-09-17T23:41:27.189326+0000","last_undegraded":"2024-09-17T23:41:27.189326+0000","last_fullsized":"2024-09-17T23:41:27.189326+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"466'99","last_scrub_stamp":"2024-09-17T23:41:27.189266+0000","last_deep_scrub":"466'99","last_deep_scrub_stamp":"2024-09-17T23:41:27.189266+0000","last_clean_scrub_stamp":"2024-09-17T23:41:27.189266+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T01:34:31.185584+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":5796,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738820,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":151472,"kb_used_data":3744,"kb_used_omap":1,"kb_used_meta":147710,"kb_avail":94220368,"statfs":{"total":96636764160,"available":96481656832,"internally_reserved":0,"allocated":3833856,"data_stored":2049348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":151255493},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738822,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":143212,"kb_used_data":3804,"kb_used_omap":1,"kb_used_meta":139390,"kb_avail":94228628,"statfs":{"total":96636764160,"available":96490115072,"internally_reserved":0,"allocated":3895296,"data_stored":2639735,"data_compressed":2898,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":142735813},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738820,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":157164,"kb_used_data":3804,"kb_used_omap":1,"kb_used_meta":153342,"kb_avail":94214676,"statfs":{"total":96636764160,"available":96475828224,"internally_reserved":0,"allocated":3895296,"data_stored":2639735,"data_compressed":2898,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":157022661},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2898,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2898,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T23:41:47.753 DEBUG:teuthology.orchestra.run.smithi006:> sudo ceph --cluster cluster1 config set global mon_health_to_clog false 2024-09-17T23:41:48.059 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-17T23:41:48.059 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-17T23:41:48.059 DEBUG:tasks.ceph.cluster1.osd.0:waiting for process to exit 2024-09-17T23:41:48.059 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T23:41:48.097 INFO:tasks.ceph.cluster1.osd.0:Stopped 2024-09-17T23:41:48.097 DEBUG:tasks.ceph.cluster1.osd.1:waiting for process to exit 2024-09-17T23:41:48.097 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T23:41:48.178 INFO:tasks.ceph.cluster1.osd.1:Stopped 2024-09-17T23:41:48.178 DEBUG:tasks.ceph.cluster1.osd.2:waiting for process to exit 2024-09-17T23:41:48.179 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T23:41:48.227 INFO:tasks.ceph.cluster1.osd.2:Stopped 2024-09-17T23:41:48.228 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-17T23:41:48.228 DEBUG:tasks.ceph.cluster1.mgr.x:waiting for process to exit 2024-09-17T23:41:48.228 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T23:41:48.254 INFO:tasks.ceph.cluster1.mgr.x:Stopped 2024-09-17T23:41:48.254 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-17T23:41:48.254 DEBUG:tasks.ceph.cluster1.mon.a:waiting for process to exit 2024-09-17T23:41:48.254 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T23:41:48.306 INFO:tasks.ceph.cluster1.mon.a:Stopped 2024-09-17T23:41:48.306 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-17T23:41:48.306 DEBUG:teuthology.orchestra.run.smithi006:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/cluster1.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2024-09-17T23:41:48.374 INFO:teuthology.orchestra.run.smithi006.stdout:2024-09-17T23:07:26.087572+0000 mon.a (mon.0) 208 : cluster [WRN] Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-17T23:41:48.374 WARNING:tasks.ceph:Found errors (ERR|WRN|SEC) in cluster log 2024-09-17T23:41:48.374 DEBUG:teuthology.orchestra.run.smithi006:> sudo egrep '\[SEC\]' /var/log/ceph/cluster1.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2024-09-17T23:41:48.441 DEBUG:teuthology.orchestra.run.smithi006:> sudo egrep '\[ERR\]' /var/log/ceph/cluster1.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2024-09-17T23:41:48.508 DEBUG:teuthology.orchestra.run.smithi006:> sudo egrep '\[WRN\]' /var/log/ceph/cluster1.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2024-09-17T23:41:48.574 INFO:teuthology.orchestra.run.smithi006.stdout:2024-09-17T23:07:26.087572+0000 mon.a (mon.0) 208 : cluster [WRN] Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-17T23:41:48.575 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-0 on ubuntu@smithi006.front.sepia.ceph.com 2024-09-17T23:41:48.575 DEBUG:teuthology.orchestra.run.smithi006:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-0 2024-09-17T23:41:48.774 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-1 on ubuntu@smithi006.front.sepia.ceph.com 2024-09-17T23:41:48.775 DEBUG:teuthology.orchestra.run.smithi006:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-1 2024-09-17T23:41:48.857 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-2 on ubuntu@smithi006.front.sepia.ceph.com 2024-09-17T23:41:48.858 DEBUG:teuthology.orchestra.run.smithi006:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-2 2024-09-17T23:41:48.922 INFO:tasks.ceph:Archiving mon data... 2024-09-17T23:41:48.923 DEBUG:teuthology.misc:Transferring archived files from smithi006:/var/lib/ceph/mon/cluster1-a to /home/teuthworker/archive/teuthology-2024-09-17_21:16:02-rbd-squid-distro-default-smithi/7909572/data/cluster1.mon.a.tgz 2024-09-17T23:41:48.923 DEBUG:teuthology.orchestra.run.smithi006:> mktemp 2024-09-17T23:41:48.942 INFO:teuthology.orchestra.run.smithi006.stdout:/tmp/tmp.MKe9mfBspl 2024-09-17T23:41:48.942 DEBUG:teuthology.orchestra.run.smithi006:> sudo tar cz -f - -C /var/lib/ceph/mon/cluster1-a -- . > /tmp/tmp.MKe9mfBspl 2024-09-17T23:41:49.104 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 0666 /tmp/tmp.MKe9mfBspl 2024-09-17T23:41:49.189 DEBUG:teuthology.orchestra.remote:smithi006:/tmp/tmp.MKe9mfBspl is 394KB 2024-09-17T23:41:49.286 DEBUG:teuthology.orchestra.run.smithi006:> rm -fr /tmp/tmp.MKe9mfBspl 2024-09-17T23:41:49.301 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-17T23:41:49.302 DEBUG:teuthology.orchestra.run.smithi006:> sudo rm -rf -- /etc/ceph/cluster1.conf /etc/ceph/cluster1.keyring /home/ubuntu/cephtest/cluster1.data /home/ubuntu/cephtest/cluster1.monmap /home/ubuntu/cephtest/../*.pid 2024-09-17T23:41:49.343 DEBUG:teuthology.orchestra.run.smithi038:> sudo rm -rf -- /etc/ceph/cluster1.conf /etc/ceph/cluster1.keyring /home/ubuntu/cephtest/cluster1.data /home/ubuntu/cephtest/cluster1.monmap /home/ubuntu/cephtest/../*.pid 2024-09-17T23:41:49.422 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-17T23:41:49.452 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-17T23:41:49.452 INFO:tasks.ceph:Archiving crash dumps... 2024-09-17T23:41:49.454 DEBUG:teuthology.misc:Transferring archived files from smithi006:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-17_21:16:02-rbd-squid-distro-default-smithi/7909572/remote/smithi006/crash 2024-09-17T23:41:49.455 DEBUG:teuthology.orchestra.run.smithi006:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-17T23:41:49.487 DEBUG:teuthology.misc:Transferring archived files from smithi038:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-17_21:16:02-rbd-squid-distro-default-smithi/7909572/remote/smithi038/crash 2024-09-17T23:41:49.488 DEBUG:teuthology.orchestra.run.smithi038:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-17T23:41:49.520 INFO:tasks.ceph:Compressing logs... 2024-09-17T23:41:49.520 DEBUG:teuthology.orchestra.run.smithi006:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-17T23:41:49.531 DEBUG:teuthology.orchestra.run.smithi038:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-17T23:41:49.561 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35981.log 2024-09-17T23:41:49.561 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37365.log 2024-09-17T23:41:49.561 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35477.log 2024-09-17T23:41:49.562 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/cluster1-client.admin.35981.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35981.log.gz 2024-09-17T23:41:49.562 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/cluster1-client.admin.37365.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35855.log 2024-09-17T23:41:49.562 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37365.log.gz 2024-09-17T23:41:49.562 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/cluster1-client.admin.35477.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.32651.log 2024-09-17T23:41:49.562 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35477.log.gz 2024-09-17T23:41:49.562 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/cluster1-client.admin.35855.log: gzip -5 --verbose -- /var/log/ceph/cluster1-mon.a.log 2024-09-17T23:41:49.562 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35855.log.gz 2024-09-17T23:41:49.562 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/cluster1-client.admin.32651.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36065.log 2024-09-17T23:41:49.562 INFO:teuthology.orchestra.run.smithi006.stderr: 25.8% -- replaced with /var/log/ceph/cluster1-client.admin.32651.log.gz 2024-09-17T23:41:49.562 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/cluster1-mon.a.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35769.log 2024-09-17T23:41:49.562 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/cluster1-client.admin.36065.log: gzip -5 --verbose -- /var/log/ceph/cluster1-osd.1.log 2024-09-17T23:41:49.562 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36065.log.gz 2024-09-17T23:41:49.562 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/cluster1-client.admin.35769.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35519.log 2024-09-17T23:41:49.562 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35769.log.gz 2024-09-17T23:41:49.563 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/cluster1-osd.1.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.32229.log 2024-09-17T23:41:49.563 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/cluster1-client.admin.35519.log: gzip -5 0.0% --verbose -- replaced with /var/log/ceph/cluster1-client.admin.35519.log.gz -- 2024-09-17T23:41:49.563 INFO:teuthology.orchestra.run.smithi006.stderr: /var/log/ceph/cluster1-client.admin.35915.log 2024-09-17T23:41:49.563 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/cluster1-client.admin.32229.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.32476.log 2024-09-17T23:41:49.563 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.32229.log.gz 2024-09-17T23:41:49.563 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/cluster1-client.admin.35915.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35915.log.gz 2024-09-17T23:41:49.563 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.27738.log 2024-09-17T23:41:49.563 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/cluster1-client.admin.32476.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37155.log 2024-09-17T23:41:49.563 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.32476.log.gz 2024-09-17T23:41:49.563 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph.tmp-client.admin.27738.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35729.log 2024-09-17T23:41:49.563 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.27738.log.gz 2024-09-17T23:41:49.563 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/cluster1-client.admin.37155.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37197.log 2024-09-17T23:41:49.563 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37155.log.gz 2024-09-17T23:41:49.563 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/cluster1-client.admin.35729.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35426.log 2024-09-17T23:41:49.564 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35729.log.gz 2024-09-17T23:41:49.564 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/cluster1-client.admin.37197.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37239.log 2024-09-17T23:41:49.564 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37197.log.gz 2024-09-17T23:41:49.564 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/cluster1-client.admin.35426.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37491.log 2024-09-17T23:41:49.564 INFO:teuthology.orchestra.run.smithi006.stderr: 80.1% -- replaced with /var/log/ceph/cluster1-client.admin.35426.log.gz 2024-09-17T23:41:49.564 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/cluster1-client.admin.37239.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37239.log.gz 2024-09-17T23:41:49.564 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35401.log 2024-09-17T23:41:49.564 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/cluster1-client.admin.37491.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.32431.log 2024-09-17T23:41:49.564 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37491.log.gz 2024-09-17T23:41:49.564 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/cluster1-client.admin.35401.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35401.log.gz 2024-09-17T23:41:49.564 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35875.log 2024-09-17T23:41:49.564 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/cluster1-client.admin.32431.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37533.log 2024-09-17T23:41:49.564 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.32431.log.gz 2024-09-17T23:41:49.564 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/cluster1-client.admin.35875.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37281.log 2024-09-17T23:41:49.565 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35875.log.gz 2024-09-17T23:41:49.565 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/cluster1-client.admin.37533.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37575.log 2024-09-17T23:41:49.565 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37533.log.gz 2024-09-17T23:41:49.565 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/cluster1-client.admin.37281.log: gzip -5 --verbose -- 0.0% /var/log/ceph/cluster1-osd.2.log -- replaced with /var/log/ceph/cluster1-client.admin.37281.log.gz 2024-09-17T23:41:49.565 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-17T23:41:49.565 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/cluster1-client.admin.37575.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36023.log 2024-09-17T23:41:49.565 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37575.log.gz 2024-09-17T23:41:49.565 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/cluster1-osd.2.log: gzip -5 --verbose -- /var/log/ceph/cluster2-mgr.x.log 2024-09-17T23:41:49.565 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/cluster1-client.admin.36023.log: 0.0%gzip -- replaced with /var/log/ceph/cluster1-client.admin.36023.log.gz -5 2024-09-17T23:41:49.565 INFO:teuthology.orchestra.run.smithi006.stderr: --verbose -- /var/log/ceph/cluster1-client.admin.37113.log 2024-09-17T23:41:49.575 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/cluster2-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.32388.log 2024-09-17T23:41:49.576 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/cluster1-client.admin.37113.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37113.log.gz 2024-09-17T23:41:49.576 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.32577.log 2024-09-17T23:41:49.576 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-mgr.x.log 2024-09-17T23:41:49.576 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/cluster1-client.admin.32388.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.32388.log.gz 2024-09-17T23:41:49.576 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/cluster1-client.admin.32577.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37615.log 2024-09-17T23:41:49.576 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.32577.log.gz 2024-09-17T23:41:49.576 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/cluster1-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/cluster1.audit.log 2024-09-17T23:41:49.576 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/cluster1-client.admin.37615.log: gzip 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37615.log.gz 2024-09-17T23:41:49.576 INFO:teuthology.orchestra.run.smithi006.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.admin.37407.log 2024-09-17T23:41:49.576 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/cluster1.audit.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35256.log 2024-09-17T23:41:49.576 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/cluster1-client.admin.37407.log: 0.0%gzip -- replaced with /var/log/ceph/cluster1-client.admin.37407.log.gz 2024-09-17T23:41:49.576 INFO:teuthology.orchestra.run.smithi006.stderr: -5 --verbose -- /var/log/ceph/cluster1-osd.0.log 2024-09-17T23:41:49.577 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/cluster1-client.admin.35256.log: gzip -5 --verbose 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35256.log.gz 2024-09-17T23:41:49.577 INFO:teuthology.orchestra.run.smithi006.stderr: -- /var/log/ceph/cluster1-client.admin.35643.log 2024-09-17T23:41:49.577 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/cluster1-osd.0.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35770.log 2024-09-17T23:41:49.577 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/cluster1-client.admin.35643.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35643.log.gz 2024-09-17T23:41:49.577 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35345.log 2024-09-17T23:41:49.577 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/cluster1-client.admin.35770.log: 92.9% -- replaced with /var/log/ceph/cluster1.audit.log.gz 2024-09-17T23:41:49.577 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35770.log.gz 2024-09-17T23:41:49.577 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35642.log 2024-09-17T23:41:49.577 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/cluster1-client.admin.35345.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35345.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37449.log 2024-09-17T23:41:49.577 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-17T23:41:49.577 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35561.log 2024-09-17T23:41:49.577 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/cluster1-client.admin.35642.log: /var/log/ceph/cluster1-client.admin.37449.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37449.log.gz 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35642.log.gz 2024-09-17T23:41:49.577 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-17T23:41:49.577 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35641.log 2024-09-17T23:41:49.578 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/cluster1-client.admin.35561.log: 0.0%gzip -5 --verbose -- /var/log/ceph/cluster1.log 2024-09-17T23:41:49.578 INFO:teuthology.orchestra.run.smithi006.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.35561.log.gz 2024-09-17T23:41:49.578 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/cluster1-client.admin.35641.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35641.log.gz 2024-09-17T23:41:49.578 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37323.log 2024-09-17T23:41:49.578 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/cluster1.log: /var/log/ceph/cluster1-client.admin.37323.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37323.log.gz 2024-09-17T23:41:49.578 INFO:teuthology.orchestra.run.smithi006.stderr: 88.0% -- replaced with /var/log/ceph/cluster1.log.gz 2024-09-17T23:41:49.604 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67935.log 2024-09-17T23:41:49.604 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53192.log 2024-09-17T23:41:49.605 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.67935.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39503.log 2024-09-17T23:41:49.605 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.53192.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71180.log 2024-09-17T23:41:49.606 INFO:teuthology.orchestra.run.smithi038.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.67935.log.gz 2024-09-17T23:41:49.606 INFO:teuthology.orchestra.run.smithi038.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.53192.log.gz 2024-09-17T23:41:49.606 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.39503.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48600.log 2024-09-17T23:41:49.606 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.71180.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.71180.log.gz 2024-09-17T23:41:49.606 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56882.log 2024-09-17T23:41:49.609 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.48600.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51636.log 2024-09-17T23:41:49.609 INFO:teuthology.orchestra.run.smithi038.stderr: 87.4%/var/log/ceph/cluster2-client.mirror.56882.log: -- replaced with /var/log/ceph/cluster2-client.mirror.48600.log.gz 2024-09-17T23:41:49.609 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54350.log 2024-09-17T23:41:49.609 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.51636.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.56882.log.gz 2024-09-17T23:41:49.610 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57455.log 2024-09-17T23:41:49.610 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.54350.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44437.log 2024-09-17T23:41:49.610 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.57455.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.54350.log.gz 2024-09-17T23:41:49.610 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40537.log 2024-09-17T23:41:49.610 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.44437.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69537.log 2024-09-17T23:41:49.610 INFO:teuthology.orchestra.run.smithi038.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51636.log.gz 2024-09-17T23:41:49.610 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.40537.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64439.log 2024-09-17T23:41:49.610 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.69537.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46276.log 2024-09-17T23:41:49.610 INFO:teuthology.orchestra.run.smithi038.stderr: 85.5% -- replaced with /var/log/ceph/cluster1-client.mirror.69537.log.gz 2024-09-17T23:41:49.610 INFO:teuthology.orchestra.run.smithi038.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.57455.log.gz 2024-09-17T23:41:49.610 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.64439.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51543.log 2024-09-17T23:41:49.610 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.46276.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62370.log 2024-09-17T23:41:49.610 INFO:teuthology.orchestra.run.smithi038.stderr: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.46276.log.gz 2024-09-17T23:41:49.610 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.51543.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53795.log 2024-09-17T23:41:49.621 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.62370.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.51543.log.gz 2024-09-17T23:41:49.621 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67135.log 2024-09-17T23:41:49.621 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50125.log 2024-09-17T23:41:49.621 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.53795.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.62370.log.gz 2024-09-17T23:41:49.621 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.67135.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70889.log 2024-09-17T23:41:49.621 INFO:teuthology.orchestra.run.smithi038.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.53795.log.gz 2024-09-17T23:41:49.621 INFO:teuthology.orchestra.run.smithi038.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.67135.log.gz 2024-09-17T23:41:49.621 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.50125.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.64439.log.gz 2024-09-17T23:41:49.621 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53819.log 2024-09-17T23:41:49.621 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.70889.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45785.log 2024-09-17T23:41:49.621 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.53819.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66303.log 2024-09-17T23:41:49.621 INFO:teuthology.orchestra.run.smithi038.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50125.log.gz 2024-09-17T23:41:49.622 INFO:teuthology.orchestra.run.smithi038.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.53819.log.gz 2024-09-17T23:41:49.622 INFO:teuthology.orchestra.run.smithi038.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.70889.log.gz 2024-09-17T23:41:49.622 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.45785.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63716.log 2024-09-17T23:41:49.622 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.66303.log: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.44437.log.gz 2024-09-17T23:41:49.622 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69962.log 2024-09-17T23:41:49.622 INFO:teuthology.orchestra.run.smithi038.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66303.log.gz 2024-09-17T23:41:49.622 INFO:teuthology.orchestra.run.smithi038.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.45785.log.gz 2024-09-17T23:41:49.622 INFO:teuthology.orchestra.run.smithi038.stderr:gzip/var/log/ceph/cluster2-client.mirror.63716.log: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51716.log 2024-09-17T23:41:49.622 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.69962.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.32637.log 2024-09-17T23:41:49.622 INFO:teuthology.orchestra.run.smithi038.stderr: 76.7% -- replaced with /var/log/ceph/cluster2-client.mirror.69962.log.gz 2024-09-17T23:41:49.622 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.51716.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44374.log 2024-09-17T23:41:49.622 INFO:teuthology.orchestra.run.smithi038.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.63716.log.gz 2024-09-17T23:41:49.622 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.admin.32637.log: gzip -5 87.6% --verbose -- -- replaced with /var/log/ceph/cluster1-client.mirror.39503.log.gz 2024-09-17T23:41:49.622 INFO:teuthology.orchestra.run.smithi038.stderr: /var/log/ceph/cluster2-client.mirror.63654.log 2024-09-17T23:41:49.623 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.32637.log.gz 2024-09-17T23:41:49.623 INFO:teuthology.orchestra.run.smithi038.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51716.log.gz 2024-09-17T23:41:49.623 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.44374.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42847.log 2024-09-17T23:41:49.623 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.63654.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62391.log 2024-09-17T23:41:49.623 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.42847.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63365.log 2024-09-17T23:41:49.623 INFO:teuthology.orchestra.run.smithi038.stderr: 89.1% -- replaced with /var/log/ceph/cluster1-client.mirror.42847.log.gz 2024-09-17T23:41:49.623 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.62391.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.63654.log.gz 2024-09-17T23:41:49.623 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37671.log 2024-09-17T23:41:49.623 INFO:teuthology.orchestra.run.smithi038.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.62391.log.gz 2024-09-17T23:41:49.623 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69097.log 2024-09-17T23:41:49.623 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.63365.log: /var/log/ceph/cluster2-client.mirror.37671.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50043.log 2024-09-17T23:41:49.623 INFO:teuthology.orchestra.run.smithi038.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.63365.log.gz 2024-09-17T23:41:49.623 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.69097.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47778.log 2024-09-17T23:41:49.623 INFO:teuthology.orchestra.run.smithi038.stderr: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.69097.log.gz 2024-09-17T23:41:49.624 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.50043.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40009.log 2024-09-17T23:41:49.624 INFO:teuthology.orchestra.run.smithi038.stderr: 1.3% -- replaced with /var/log/ceph/cluster1-client.mirror.50043.log.gz 2024-09-17T23:41:49.624 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.47778.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67849.log 2024-09-17T23:41:49.624 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.40009.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62016.log 2024-09-17T23:41:49.624 INFO:teuthology.orchestra.run.smithi038.stderr: 54.5% -- replaced with /var/log/ceph/cluster2-client.mirror.37671.log.gz 2024-09-17T23:41:49.624 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62742.log 2024-09-17T23:41:49.624 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.67849.log: /var/log/ceph/cluster2-client.mirror.62016.log: gzip 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.67849.log.gz 2024-09-17T23:41:49.624 INFO:teuthology.orchestra.run.smithi038.stderr: -5 --verbose 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.40009.log.gz 2024-09-17T23:41:49.624 INFO:teuthology.orchestra.run.smithi038.stderr: -- /var/log/ceph/cluster1-client.mirror.57019.log 2024-09-17T23:41:49.624 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.62742.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67293.log 2024-09-17T23:41:49.624 INFO:teuthology.orchestra.run.smithi038.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.62016.log.gz 2024-09-17T23:41:49.624 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.57019.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57906.log 2024-09-17T23:41:49.624 INFO:teuthology.orchestra.run.smithi038.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.44374.log.gz 2024-09-17T23:41:49.624 INFO:teuthology.orchestra.run.smithi038.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.62742.log.gz 2024-09-17T23:41:49.625 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.67293.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36350.log 2024-09-17T23:41:49.625 INFO:teuthology.orchestra.run.smithi038.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67293.log.gz 2024-09-17T23:41:49.625 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39183.log 2024-09-17T23:41:49.625 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.57906.log: /var/log/ceph/cluster1-client.admin.36350.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36350.log.gz 2024-09-17T23:41:49.625 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46415.log 2024-09-17T23:41:49.625 INFO:teuthology.orchestra.run.smithi038.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.57906.log.gz 2024-09-17T23:41:49.625 INFO:teuthology.orchestra.run.smithi038.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.57019.log.gz 2024-09-17T23:41:49.625 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54630.log 2024-09-17T23:41:49.625 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.39183.log: /var/log/ceph/cluster2-client.mirror.46415.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51832.log 2024-09-17T23:41:49.625 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.54630.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59655.log 2024-09-17T23:41:49.625 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.51832.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64481.log 2024-09-17T23:41:49.625 INFO:teuthology.orchestra.run.smithi038.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.39183.log.gz 2024-09-17T23:41:49.647 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.59655.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52801.log 2024-09-17T23:41:49.648 INFO:teuthology.orchestra.run.smithi038.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.54630.log.gz 2024-09-17T23:41:49.648 INFO:teuthology.orchestra.run.smithi038.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.51832.log.gz 2024-09-17T23:41:49.648 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.64481.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56954.log 2024-09-17T23:41:49.648 INFO:teuthology.orchestra.run.smithi038.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.59655.log.gz 2024-09-17T23:41:49.648 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.52801.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36266.log 2024-09-17T23:41:49.648 INFO:teuthology.orchestra.run.smithi038.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.64481.log.gz 2024-09-17T23:41:49.648 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.56954.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.62156.log 2024-09-17T23:41:49.648 INFO:teuthology.orchestra.run.smithi038.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.40537.log.gz 2024-09-17T23:41:49.648 INFO:teuthology.orchestra.run.smithi038.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.52801.log.gz 2024-09-17T23:41:49.648 INFO:teuthology.orchestra.run.smithi038.stderr: 87.7%/var/log/ceph/cluster1-client.admin.36266.log: -- replaced with /var/log/ceph/cluster1-client.mirror.56954.log.gz 2024-09-17T23:41:49.648 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64915.log 2024-09-17T23:41:49.648 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36266.log.gz 2024-09-17T23:41:49.649 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.1.62156.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58147.log 2024-09-17T23:41:49.649 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.64915.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56368.log 2024-09-17T23:41:49.649 INFO:teuthology.orchestra.run.smithi038.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.64915.log.gz 2024-09-17T23:41:49.649 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.58147.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56906.log 2024-09-17T23:41:49.649 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.56368.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.58147.log.gz 2024-09-17T23:41:49.649 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41046.log 2024-09-17T23:41:49.649 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.56906.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65088.log 2024-09-17T23:41:49.649 INFO:teuthology.orchestra.run.smithi038.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.56906.log.gz 2024-09-17T23:41:49.649 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.41046.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53216.log 2024-09-17T23:41:49.649 INFO:teuthology.orchestra.run.smithi038.stderr: 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.1.62156.log.gz 2024-09-17T23:41:49.649 INFO:teuthology.orchestra.run.smithi038.stderr: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.56368.log.gz 2024-09-17T23:41:49.649 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose/var/log/ceph/cluster1-client.mirror.65088.log: -- /var/log/ceph/cluster1-client.mirror.0.68453.log 2024-09-17T23:41:49.649 INFO:teuthology.orchestra.run.smithi038.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.65088.log.gz 2024-09-17T23:41:49.650 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.53216.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53498.log 2024-09-17T23:41:49.650 INFO:teuthology.orchestra.run.smithi038.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.53216.log.gz 2024-09-17T23:41:49.650 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50699.log 2024-09-17T23:41:49.650 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.0.68453.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62246.log 2024-09-17T23:41:49.650 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.53498.log: /var/log/ceph/cluster1-client.mirror.50699.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57258.log 2024-09-17T23:41:49.650 INFO:teuthology.orchestra.run.smithi038.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.50699.log.gz 2024-09-17T23:41:49.650 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.62246.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48966.log 2024-09-17T23:41:49.650 INFO:teuthology.orchestra.run.smithi038.stderr:gzip/var/log/ceph/cluster2-client.mirror.57258.log: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65875.log 2024-09-17T23:41:49.650 INFO:teuthology.orchestra.run.smithi038.stderr: 89.4%/var/log/ceph/cluster2-client.mirror.48966.log: -- replaced with /var/log/ceph/cluster2-client.mirror.62246.log.gz 2024-09-17T23:41:49.650 INFO:teuthology.orchestra.run.smithi038.stderr: 92.4%gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.32325.log 2024-09-17T23:41:49.651 INFO:teuthology.orchestra.run.smithi038.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.53498.log.gz 2024-09-17T23:41:49.651 INFO:teuthology.orchestra.run.smithi038.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.57258.log.gz 2024-09-17T23:41:49.651 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.65875.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60262.log 2024-09-17T23:41:49.651 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.admin.32325.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.65875.log.gz 2024-09-17T23:41:49.651 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.57732.log 2024-09-17T23:41:49.651 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.32325.log.gz 2024-09-17T23:41:49.651 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61766.log 2024-09-17T23:41:49.651 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.60262.log: /var/log/ceph/cluster2-client.mirror.0.57732.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58019.log 2024-09-17T23:41:49.651 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.61766.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.60262.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40842.log 2024-09-17T23:41:49.651 INFO:teuthology.orchestra.run.smithi038.stderr: 2024-09-17T23:41:49.651 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.61766.log.gz 2024-09-17T23:41:49.651 INFO:teuthology.orchestra.run.smithi038.stderr: /var/log/ceph/cluster1-client.mirror.65133.log 2024-09-17T23:41:49.651 INFO:teuthology.orchestra.run.smithi038.stderr: 93.6% -- replaced with /var/log/ceph/cluster1-client.mirror.0.68453.log.gz 2024-09-17T23:41:49.651 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.40842.log: /var/log/ceph/cluster1-client.mirror.58019.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40237.log 2024-09-17T23:41:49.652 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.65133.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40121.log 2024-09-17T23:41:49.652 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.admin.40237.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43528.log 2024-09-17T23:41:49.652 INFO:teuthology.orchestra.run.smithi038.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.40842.log.gz/var/log/ceph/cluster2-client.mirror.40121.log: 2024-09-17T23:41:49.652 INFO:teuthology.orchestra.run.smithi038.stderr: 92.8% -- replaced with /var/log/ceph/cluster2-client.mirror.0.57732.log.gz 2024-09-17T23:41:49.652 INFO:teuthology.orchestra.run.smithi038.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.65133.log.gz 2024-09-17T23:41:49.652 INFO:teuthology.orchestra.run.smithi038.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.40237.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56930.log 2024-09-17T23:41:49.652 INFO:teuthology.orchestra.run.smithi038.stderr: 2024-09-17T23:41:49.652 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.43528.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36098.log 2024-09-17T23:41:49.652 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.56930.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47086.log 2024-09-17T23:41:49.652 INFO:teuthology.orchestra.run.smithi038.stderr: 87.4%/var/log/ceph/cluster2-client.admin.36098.log: -- replaced with /var/log/ceph/cluster1-client.mirror.56930.log.gz 2024-09-17T23:41:49.652 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36098.log.gz 2024-09-17T23:41:49.652 INFO:teuthology.orchestra.run.smithi038.stderr: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.43528.log.gz 2024-09-17T23:41:49.652 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.71823.log 2024-09-17T23:41:49.653 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56279.log 2024-09-17T23:41:49.653 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.47086.log: /var/log/ceph/cluster2-client.admin.71823.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47086.log.gz 2024-09-17T23:41:49.653 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66505.log 2024-09-17T23:41:49.653 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.71823.log.gz 2024-09-17T23:41:49.653 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44652.log 2024-09-17T23:41:49.653 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.56279.log: /var/log/ceph/cluster1-client.mirror.66505.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48172.log 2024-09-17T23:41:49.653 INFO:teuthology.orchestra.run.smithi038.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.56279.log.gz 2024-09-17T23:41:49.653 INFO:teuthology.orchestra.run.smithi038.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66505.log.gz 2024-09-17T23:41:49.653 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47350.log 2024-09-17T23:41:49.653 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.44652.log: /var/log/ceph/cluster2-client.mirror.48172.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47189.log 2024-09-17T23:41:49.653 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.47350.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.44652.log.gz 2024-09-17T23:41:49.653 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39798.log 2024-09-17T23:41:49.653 INFO:teuthology.orchestra.run.smithi038.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48172.log.gz 2024-09-17T23:41:49.653 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.47189.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39455.log 2024-09-17T23:41:49.654 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.39798.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59587.log 2024-09-17T23:41:49.654 INFO:teuthology.orchestra.run.smithi038.stderr: 85.0% -- replaced with /var/log/ceph/cluster1-client.mirror.39798.log.gz 2024-09-17T23:41:49.654 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.39455.log: 93.3%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65380.log 2024-09-17T23:41:49.654 INFO:teuthology.orchestra.run.smithi038.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.41046.log.gz 2024-09-17T23:41:49.654 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.59587.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.39455.log.gz 2024-09-17T23:41:49.654 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43795.log 2024-09-17T23:41:49.654 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.65380.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.59587.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52676.log 2024-09-17T23:41:49.654 INFO:teuthology.orchestra.run.smithi038.stderr: 2024-09-17T23:41:49.654 INFO:teuthology.orchestra.run.smithi038.stderr: 85.0% -- replaced with /var/log/ceph/cluster1-client.mirror.65380.log.gz 2024-09-17T23:41:49.654 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.43795.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61249.log 2024-09-17T23:41:49.654 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.52676.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43548.log 2024-09-17T23:41:49.654 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.61249.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.52676.log.gz 2024-09-17T23:41:49.654 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46373.log 2024-09-17T23:41:49.654 INFO:teuthology.orchestra.run.smithi038.stderr: 89.6%/var/log/ceph/cluster2-client.mirror.43548.log: -- replaced with /var/log/ceph/cluster2-client.mirror.61249.log.gz 2024-09-17T23:41:49.655 INFO:teuthology.orchestra.run.smithi038.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.43548.log.gz 2024-09-17T23:41:49.655 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41467.log 2024-09-17T23:41:49.655 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.46373.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47615.log 2024-09-17T23:41:49.655 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.41467.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.46373.log.gz 2024-09-17T23:41:49.655 INFO:teuthology.orchestra.run.smithi038.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.41467.log.gz 2024-09-17T23:41:49.655 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.3.68624.log 2024-09-17T23:41:49.655 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.47615.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40141.log 2024-09-17T23:41:49.655 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.3.68624.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.58019.log.gz 2024-09-17T23:41:49.658 INFO:teuthology.orchestra.run.smithi038.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.43795.log.gz 2024-09-17T23:41:49.658 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37406.log 2024-09-17T23:41:49.658 INFO:teuthology.orchestra.run.smithi038.stderr: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.3.68624.log.gz 2024-09-17T23:41:49.658 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.40141.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41347.log 2024-09-17T23:41:49.658 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67554.log 2024-09-17T23:41:49.658 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.admin.37406.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37406.log.gz 2024-09-17T23:41:49.658 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.41347.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43881.log 2024-09-17T23:41:49.659 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.67554.log: 86.6% -- replaced with /var/log/ceph/cluster2-client.mirror.67554.log.gz 2024-09-17T23:41:49.659 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.43881.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63179.log 2024-09-17T23:41:49.661 INFO:teuthology.orchestra.run.smithi038.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.43881.log.gz 2024-09-17T23:41:49.663 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.63179.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.40121.log.gz 2024-09-17T23:41:49.663 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69738.log 2024-09-17T23:41:49.663 INFO:teuthology.orchestra.run.smithi038.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.63179.log.gz 2024-09-17T23:41:49.664 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43076.log 2024-09-17T23:41:49.664 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65943.log 2024-09-17T23:41:49.664 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.43076.log: /var/log/ceph/cluster2-client.mirror.69738.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.69738.log.gz 2024-09-17T23:41:49.666 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66753.log 2024-09-17T23:41:49.666 INFO:teuthology.orchestra.run.smithi038.stderr: 91.6%/var/log/ceph/cluster1-client.mirror.65943.log: -- replaced with /var/log/ceph/cluster2-client.mirror.43076.log.gz 2024-09-17T23:41:49.666 INFO:teuthology.orchestra.run.smithi038.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.65943.log.gz 2024-09-17T23:41:49.666 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50499.log 2024-09-17T23:41:49.666 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.66753.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.36850.log 2024-09-17T23:41:49.667 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.50499.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.50499.log.gz 2024-09-17T23:41:49.671 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54905.log 2024-09-17T23:41:49.671 INFO:teuthology.orchestra.run.smithi038.stderr: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.40141.log.gz 2024-09-17T23:41:49.671 INFO:teuthology.orchestra.run.smithi038.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66753.log.gz 2024-09-17T23:41:49.671 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.1.36850.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58629.log 2024-09-17T23:41:49.671 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.54905.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70494.log 2024-09-17T23:41:49.671 INFO:teuthology.orchestra.run.smithi038.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.54905.log.gz 2024-09-17T23:41:49.671 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.58629.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51374.log 2024-09-17T23:41:49.671 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.70494.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.58629.log.gz 2024-09-17T23:41:49.673 INFO:teuthology.orchestra.run.smithi038.stderr: 91.1% -- replaced with /var/log/ceph/cluster2-client.mirror.70494.log.gz 2024-09-17T23:41:49.679 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66258.log 2024-09-17T23:41:49.679 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.51374.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38698.log 2024-09-17T23:41:49.679 INFO:teuthology.orchestra.run.smithi038.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.51374.log.gz 2024-09-17T23:41:49.679 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.66258.log: 86.8%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55845.log 2024-09-17T23:41:49.679 INFO:teuthology.orchestra.run.smithi038.stderr: 88.5% -- replaced with /var/log/ceph/cluster1-client.mirror.66258.log.gz 2024-09-17T23:41:49.679 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.admin.38698.log: -- replaced with /var/log/ceph/cluster2-client.mirror.41347.log.gz 2024-09-17T23:41:49.679 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55652.log 2024-09-17T23:41:49.679 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.55845.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60690.log 2024-09-17T23:41:49.679 INFO:teuthology.orchestra.run.smithi038.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.55845.log.gz 2024-09-17T23:41:49.679 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.55652.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70766.log 2024-09-17T23:41:49.679 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.60690.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.38698.log.gz 2024-09-17T23:41:49.679 INFO:teuthology.orchestra.run.smithi038.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.55652.log.gz 2024-09-17T23:41:49.680 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66865.log 2024-09-17T23:41:49.680 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.70766.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.70766.log.gz 2024-09-17T23:41:49.680 INFO:teuthology.orchestra.run.smithi038.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.60690.log.gz 2024-09-17T23:41:49.680 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36697.log 2024-09-17T23:41:49.693 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52628.log 2024-09-17T23:41:49.693 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.36697.log: /var/log/ceph/cluster1-client.mirror.66865.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.66865.log.gz 2024-09-17T23:41:49.694 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.72033.log 2024-09-17T23:41:49.694 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose 91.4% -- /var/log/ceph/cluster2-client.mirror.46313.log 2024-09-17T23:41:49.694 INFO:teuthology.orchestra.run.smithi038.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.36697.log.gz 2024-09-17T23:41:49.694 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.admin.72033.log: /var/log/ceph/cluster1-client.mirror.52628.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.72033.log.gz 2024-09-17T23:41:49.694 INFO:teuthology.orchestra.run.smithi038.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.52628.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35972.log 2024-09-17T23:41:49.694 INFO:teuthology.orchestra.run.smithi038.stderr: 2024-09-17T23:41:49.694 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64977.log 2024-09-17T23:41:49.694 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.46313.log: /var/log/ceph/cluster2-client.admin.35972.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.35972.log.gz 2024-09-17T23:41:49.694 INFO:teuthology.orchestra.run.smithi038.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46313.log.gz 2024-09-17T23:41:49.694 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63096.log 2024-09-17T23:41:49.694 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.64977.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65245.log 2024-09-17T23:41:49.694 INFO:teuthology.orchestra.run.smithi038.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.64977.log.gz 2024-09-17T23:41:49.694 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.63096.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46037.log 2024-09-17T23:41:49.694 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.65245.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.65245.log.gz 2024-09-17T23:41:49.695 INFO:teuthology.orchestra.run.smithi038.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.63096.log.gz 2024-09-17T23:41:49.695 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69134.log 2024-09-17T23:41:49.695 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.46037.log: gzip -5 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46037.log.gz 2024-09-17T23:41:49.695 INFO:teuthology.orchestra.run.smithi038.stderr: --verbose -- /var/log/ceph/cluster2-client.mirror.53558.log 2024-09-17T23:41:49.695 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.69134.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.69134.log.gz 2024-09-17T23:41:49.695 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44546.log 2024-09-17T23:41:49.695 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.53558.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62308.log 2024-09-17T23:41:49.695 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.44546.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.53558.log.gz 2024-09-17T23:41:49.695 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59812.log 2024-09-17T23:41:49.695 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.62308.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.62308.log.gz 2024-09-17T23:41:49.695 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48112.log 2024-09-17T23:41:49.695 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.59812.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.59812.log.gz 2024-09-17T23:41:49.695 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52266.log 2024-09-17T23:41:49.695 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.48112.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48112.log.gz 2024-09-17T23:41:49.696 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68185.log 2024-09-17T23:41:49.697 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.52266.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.44546.log.gz 2024-09-17T23:41:49.697 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65178.log 2024-09-17T23:41:49.697 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.68185.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.68185.log.gz 2024-09-17T23:41:49.697 INFO:teuthology.orchestra.run.smithi038.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.52266.log.gz 2024-09-17T23:41:49.697 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45426.log 2024-09-17T23:41:49.697 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58260.log 2024-09-17T23:41:49.698 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.65178.log: /var/log/ceph/cluster2-client.mirror.45426.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.65178.log.gz 2024-09-17T23:41:49.698 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35392.log 2024-09-17T23:41:49.700 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.58260.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.58260.log.gz 2024-09-17T23:41:49.700 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64625.log 2024-09-17T23:41:49.701 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.admin.35392.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.35392.log.gz 2024-09-17T23:41:49.702 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39224.log 2024-09-17T23:41:49.704 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.64625.log: 90.2% -- replaced with /var/log/ceph/cluster2-client.mirror.64625.log.gz 2024-09-17T23:41:49.706 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose 92.8% -- replaced with /var/log/ceph/cluster2-client.mirror.1.36850.log.gz 2024-09-17T23:41:49.706 INFO:teuthology.orchestra.run.smithi038.stderr: -- /var/log/ceph/cluster2-client.mirror.0.40701.log 2024-09-17T23:41:49.706 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.39224.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53715.log 2024-09-17T23:41:49.707 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.0.40701.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61296.log 2024-09-17T23:41:49.709 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.53715.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.53715.log.gz 2024-09-17T23:41:49.710 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65425.log 2024-09-17T23:41:49.712 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.61296.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.61296.log.gz 2024-09-17T23:41:49.712 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59766.log 2024-09-17T23:41:49.712 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.65425.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.65425.log.gz 2024-09-17T23:41:49.715 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58843.log 2024-09-17T23:41:49.719 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.59766.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.59766.log.gz 2024-09-17T23:41:49.719 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70639.log 2024-09-17T23:41:49.722 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.58843.log: 91.9% -- replaced with /var/log/ceph/cluster2-client.mirror.58843.log.gz 2024-09-17T23:41:49.723 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43944.log 2024-09-17T23:41:49.725 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.70639.log: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.70639.log.gz 2024-09-17T23:41:49.725 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57296.log 2024-09-17T23:41:49.732 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.43944.log: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.39224.log.gz 2024-09-17T23:41:49.732 INFO:teuthology.orchestra.run.smithi038.stderr: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.43944.log.gz 2024-09-17T23:41:49.733 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65044.log 2024-09-17T23:41:49.735 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.57296.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.57296.log.gz 2024-09-17T23:41:49.735 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60423.log 2024-09-17T23:41:49.736 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.65044.log: 1.3% -- replaced with /var/log/ceph/cluster1-client.mirror.65044.log.gz 2024-09-17T23:41:49.736 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.49284.log 2024-09-17T23:41:49.740 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.60423.log: 93.3% -- replaced with /var/log/ceph/cluster2-client.mirror.0.40701.log.gz 2024-09-17T23:41:49.741 INFO:teuthology.orchestra.run.smithi038.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.60423.log.gz 2024-09-17T23:41:49.742 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68014.log 2024-09-17T23:41:49.747 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.0.49284.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.0.49284.log.gz 2024-09-17T23:41:49.748 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56472.log 2024-09-17T23:41:49.749 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.68014.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.68014.log.gz 2024-09-17T23:41:49.750 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67334.log 2024-09-17T23:41:49.751 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.56472.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.56472.log.gz 2024-09-17T23:41:49.751 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45053.log 2024-09-17T23:41:49.752 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.67334.log: 1.3% -- replaced with /var/log/ceph/cluster2-client.mirror.67334.log.gz 2024-09-17T23:41:49.752 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48273.log 2024-09-17T23:41:49.756 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.45053.log: 90.9% -- replaced with /var/log/ceph/cluster2-client.mirror.45053.log.gz 2024-09-17T23:41:49.756 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51181.log 2024-09-17T23:41:49.756 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.48273.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48273.log.gz 2024-09-17T23:41:49.757 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53281.log 2024-09-17T23:41:49.758 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.51181.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.51181.log.gz 2024-09-17T23:41:49.759 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.72157.log 2024-09-17T23:41:49.761 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.53281.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.53281.log.gz 2024-09-17T23:41:49.761 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41251.log 2024-09-17T23:41:49.762 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.admin.72157.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.72157.log.gz 2024-09-17T23:41:49.762 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62037.log 2024-09-17T23:41:49.764 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.admin.41251.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.41251.log.gz 2024-09-17T23:41:49.764 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.32538.log 2024-09-17T23:41:49.765 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.62037.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.62037.log.gz 2024-09-17T23:41:49.765 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68624.log 2024-09-17T23:41:49.766 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.admin.32538.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.32538.log.gz 2024-09-17T23:41:49.767 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58099.log 2024-09-17T23:41:49.768 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.68624.log: 77.6% -- replaced with /var/log/ceph/cluster1-client.mirror.68624.log.gz 2024-09-17T23:41:49.768 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49431.log 2024-09-17T23:41:49.768 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.58099.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.58099.log.gz 2024-09-17T23:41:49.769 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50521.log 2024-09-17T23:41:49.770 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.49431.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.49431.log.gz 2024-09-17T23:41:49.771 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.32701.log 2024-09-17T23:41:49.772 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.50521.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50521.log.gz 2024-09-17T23:41:49.774 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60915.log 2024-09-17T23:41:49.774 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.admin.32701.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.32701.log.gz 2024-09-17T23:41:49.774 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54608.log 2024-09-17T23:41:49.775 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.60915.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.60915.log.gz 2024-09-17T23:41:49.776 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63861.log 2024-09-17T23:41:49.778 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.54608.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.54608.log.gz 2024-09-17T23:41:49.778 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63509.log 2024-09-17T23:41:49.779 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.63861.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.63861.log.gz 2024-09-17T23:41:49.780 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64563.log 2024-09-17T23:41:49.782 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.63509.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.63509.log.gz 2024-09-17T23:41:49.783 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46474.log 2024-09-17T23:41:49.787 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.64563.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.64563.log.gz 2024-09-17T23:41:49.787 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35468.log 2024-09-17T23:41:49.789 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.46474.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46474.log.gz 2024-09-17T23:41:49.789 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37168.log 2024-09-17T23:41:49.789 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.admin.35468.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.35468.log.gz 2024-09-17T23:41:49.791 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39361.log 2024-09-17T23:41:49.792 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.admin.37168.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37168.log.gz 2024-09-17T23:41:49.792 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63406.log 2024-09-17T23:41:49.792 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.39361.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71061.log 2024-09-17T23:41:49.792 INFO:teuthology.orchestra.run.smithi038.stderr: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.39361.log.gz 2024-09-17T23:41:49.794 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.63406.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.63406.log.gz 2024-09-17T23:41:49.797 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62907.log 2024-09-17T23:41:49.797 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.71061.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.71061.log.gz 2024-09-17T23:41:49.797 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53409.log 2024-09-17T23:41:49.799 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.62907.log: 90.2% -- replaced with /var/log/ceph/cluster2-client.mirror.62907.log.gz 2024-09-17T23:41:49.800 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48622.log 2024-09-17T23:41:49.801 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.53409.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.53409.log.gz 2024-09-17T23:41:49.801 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61207.log 2024-09-17T23:41:49.813 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.48622.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65853.log 2024-09-17T23:41:49.815 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.61207.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.61207.log.gz 2024-09-17T23:41:49.828 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40890.log 2024-09-17T23:41:49.829 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.65853.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.65853.log.gz 2024-09-17T23:41:49.841 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65988.log 2024-09-17T23:41:49.842 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.40890.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.40890.log.gz 2024-09-17T23:41:49.850 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45288.log 2024-09-17T23:41:49.850 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.65988.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.65988.log.gz 2024-09-17T23:41:49.866 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43815.log 2024-09-17T23:41:49.867 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.45288.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45288.log.gz 2024-09-17T23:41:49.871 INFO:teuthology.orchestra.run.smithi006.stderr: 91.2% -- replaced with /var/log/ceph/cluster1-mgr.x.log.gz 2024-09-17T23:41:49.876 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48092.log 2024-09-17T23:41:49.877 INFO:teuthology.orchestra.run.smithi006.stderr: 91.1% -- replaced with /var/log/ceph/cluster2-mgr.x.log.gz 2024-09-17T23:41:49.881 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.43815.log: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.43815.log.gz 2024-09-17T23:41:49.885 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58436.log 2024-09-17T23:41:49.886 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.48092.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48092.log.gz 2024-09-17T23:41:49.904 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65605.log 2024-09-17T23:41:49.905 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.58436.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.58436.log.gz 2024-09-17T23:41:49.910 INFO:teuthology.orchestra.run.smithi038.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.47189.log.gz 2024-09-17T23:41:49.911 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44282.log 2024-09-17T23:41:49.911 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.65605.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.65605.log.gz 2024-09-17T23:41:49.915 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71277.log 2024-09-17T23:41:49.915 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.44282.log: 81.2% -- replaced with /var/log/ceph/cluster1-client.mirror.44282.log.gz 2024-09-17T23:41:49.915 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49229.log 2024-09-17T23:41:49.916 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.71277.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.71277.log.gz 2024-09-17T23:41:49.916 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47719.log 2024-09-17T23:41:49.916 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.49229.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.49229.log.gz 2024-09-17T23:41:49.916 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60780.log 2024-09-17T23:41:49.916 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.47719.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.47719.log.gz 2024-09-17T23:41:49.916 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68991.log 2024-09-17T23:41:49.919 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.60780.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.60780.log.gz 2024-09-17T23:41:49.919 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37243.log 2024-09-17T23:41:49.919 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.68991.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.68991.log.gz 2024-09-17T23:41:49.919 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48659.log 2024-09-17T23:41:49.919 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.37243.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67270.log 2024-09-17T23:41:49.920 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.48659.log: 85.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48659.log.gz 2024-09-17T23:41:49.920 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44073.log 2024-09-17T23:41:49.920 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.67270.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67270.log.gz 2024-09-17T23:41:49.920 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70158.log 2024-09-17T23:41:49.925 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.44073.log: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.44073.log.gz 2024-09-17T23:41:49.927 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64109.log 2024-09-17T23:41:49.927 INFO:teuthology.orchestra.run.smithi038.stderr: 53.1% -- replaced with /var/log/ceph/cluster1-client.mirror.37243.log.gz 2024-09-17T23:41:49.927 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.70158.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.70158.log.gz 2024-09-17T23:41:49.928 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58932.log 2024-09-17T23:41:49.928 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.64109.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.64109.log.gz 2024-09-17T23:41:49.928 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58195.log 2024-09-17T23:41:49.930 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.58932.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.58932.log.gz 2024-09-17T23:41:49.930 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56428.log 2024-09-17T23:41:49.930 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.58195.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.58195.log.gz 2024-09-17T23:41:49.931 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69903.log 2024-09-17T23:41:49.931 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.56428.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.56428.log.gz 2024-09-17T23:41:49.932 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66190.log 2024-09-17T23:41:49.934 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.69903.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.69903.log.gz 2024-09-17T23:41:49.934 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49896.log 2024-09-17T23:41:49.934 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.66190.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66190.log.gz 2024-09-17T23:41:49.935 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47308.log 2024-09-17T23:41:49.944 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.49896.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49005.log 2024-09-17T23:41:49.947 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.47308.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47308.log.gz 2024-09-17T23:41:49.949 INFO:teuthology.orchestra.run.smithi038.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.47350.log.gz 2024-09-17T23:41:49.950 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69673.log 2024-09-17T23:41:49.951 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.49005.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49005.log.gz 2024-09-17T23:41:49.952 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52459.log 2024-09-17T23:41:49.954 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.69673.log: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.69673.log.gz 2024-09-17T23:41:49.954 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43987.log 2024-09-17T23:41:49.955 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.52459.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.52459.log.gz 2024-09-17T23:41:49.956 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.71432.log 2024-09-17T23:41:49.962 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.43987.log: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.43987.log.gz 2024-09-17T23:41:49.963 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42871.log 2024-09-17T23:41:49.963 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.admin.71432.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.71432.log.gz 2024-09-17T23:41:49.964 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70872.log 2024-09-17T23:41:49.964 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.42871.log: 88.3% -- replaced with /var/log/ceph/cluster1-client.mirror.42871.log.gz 2024-09-17T23:41:49.965 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49025.log 2024-09-17T23:41:49.966 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.70872.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.70872.log.gz 2024-09-17T23:41:49.966 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71040.log 2024-09-17T23:41:49.968 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.49025.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.49025.log.gz 2024-09-17T23:41:49.968 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57978.log 2024-09-17T23:41:49.969 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.71040.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.71040.log.gz 2024-09-17T23:41:49.969 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60982.log 2024-09-17T23:41:49.970 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.57978.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.57978.log.gz 2024-09-17T23:41:49.971 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42681.log 2024-09-17T23:41:49.972 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.60982.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.60982.log.gz 2024-09-17T23:41:49.972 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40581.log 2024-09-17T23:41:49.974 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.42681.log: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.42681.log.gz 2024-09-17T23:41:49.974 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46879.log 2024-09-17T23:41:49.975 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.40581.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65335.log 2024-09-17T23:41:49.977 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.46879.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46879.log.gz 2024-09-17T23:41:49.977 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66393.log 2024-09-17T23:41:49.978 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.65335.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.65335.log.gz 2024-09-17T23:41:49.979 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44852.log 2024-09-17T23:41:49.980 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.66393.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66393.log.gz 2024-09-17T23:41:49.980 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38676.log 2024-09-17T23:41:49.981 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.44852.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44852.log.gz 2024-09-17T23:41:49.982 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57236.log 2024-09-17T23:41:49.983 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.admin.38676.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61030.log 2024-09-17T23:41:49.983 INFO:teuthology.orchestra.run.smithi038.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.38676.log.gz 2024-09-17T23:41:49.984 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.57236.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.57236.log.gz 2024-09-17T23:41:49.985 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59744.log 2024-09-17T23:41:49.986 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.61030.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61030.log.gz 2024-09-17T23:41:49.987 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39135.log 2024-09-17T23:41:49.989 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.59744.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.59744.log.gz 2024-09-17T23:41:49.989 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54543.log 2024-09-17T23:41:49.991 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.39135.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.39135.log.gz 2024-09-17T23:41:49.991 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53891.log 2024-09-17T23:41:49.991 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.54543.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.54543.log.gz 2024-09-17T23:41:49.992 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49874.log 2024-09-17T23:41:49.993 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.53891.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.53891.log.gz 2024-09-17T23:41:49.994 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46694.log 2024-09-17T23:41:49.994 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.49874.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49874.log.gz 2024-09-17T23:41:49.995 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60216.log 2024-09-17T23:41:49.997 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.46694.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.46694.log.gz 2024-09-17T23:41:49.997 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.43023.log 2024-09-17T23:41:49.999 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.60216.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.60216.log.gz 2024-09-17T23:41:49.999 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65628.log 2024-09-17T23:41:49.999 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.admin.43023.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.admin.43023.log.gz 2024-09-17T23:41:50.001 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67068.log 2024-09-17T23:41:50.001 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.65628.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.65628.log.gz 2024-09-17T23:41:50.003 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37362.log 2024-09-17T23:41:50.003 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.67068.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.67068.log.gz 2024-09-17T23:41:50.003 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61316.log 2024-09-17T23:41:50.004 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.admin.37362.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37362.log.gz 2024-09-17T23:41:50.004 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43276.log 2024-09-17T23:41:50.005 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.61316.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.61316.log.gz 2024-09-17T23:41:50.005 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54885.log 2024-09-17T23:41:50.006 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.43276.log: 89.2% -- replaced with /var/log/ceph/cluster1-client.mirror.43276.log.gz 2024-09-17T23:41:50.007 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35510.log 2024-09-17T23:41:50.009 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.54885.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.54885.log.gz 2024-09-17T23:41:50.009 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37192.log 2024-09-17T23:41:50.010 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.admin.35510.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.35510.log.gz 2024-09-17T23:41:50.010 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50064.log 2024-09-17T23:41:50.011 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.admin.37192.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47228.log 2024-09-17T23:41:50.011 INFO:teuthology.orchestra.run.smithi038.stderr: 80.2% -- replaced with /var/log/ceph/cluster1-client.admin.37192.log.gz 2024-09-17T23:41:50.012 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.50064.log: 1.3% -- replaced with /var/log/ceph/cluster1-client.mirror.50064.log.gz 2024-09-17T23:41:50.023 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41926.log 2024-09-17T23:41:50.024 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.47228.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47228.log.gz 2024-09-17T23:41:50.037 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.71697.log 2024-09-17T23:41:50.039 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.41926.log: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.41926.log.gz 2024-09-17T23:41:50.052 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50892.log 2024-09-17T23:41:50.052 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.admin.71697.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.71697.log.gz 2024-09-17T23:41:50.076 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44202.log 2024-09-17T23:41:50.076 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.50892.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.50892.log.gz 2024-09-17T23:41:50.079 INFO:teuthology.orchestra.run.smithi038.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.47615.log.gz 2024-09-17T23:41:50.080 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49085.log 2024-09-17T23:41:50.086 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.44202.log: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.44202.log.gz 2024-09-17T23:41:50.087 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65313.log 2024-09-17T23:41:50.087 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.49085.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49085.log.gz 2024-09-17T23:41:50.089 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55363.log 2024-09-17T23:41:50.089 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.65313.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.65313.log.gz 2024-09-17T23:41:50.090 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54495.log 2024-09-17T23:41:50.090 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.55363.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.55363.log.gz 2024-09-17T23:41:50.091 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59362.log 2024-09-17T23:41:50.092 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.54495.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.54495.log.gz 2024-09-17T23:41:50.092 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42955.log 2024-09-17T23:41:50.092 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.59362.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.59362.log.gz 2024-09-17T23:41:50.093 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43666.log 2024-09-17T23:41:50.099 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.42955.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.42955.log.gz 2024-09-17T23:41:50.100 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56713.log 2024-09-17T23:41:50.106 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.43666.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.45426.log.gz 2024-09-17T23:41:50.106 INFO:teuthology.orchestra.run.smithi038.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.43666.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59724.log 2024-09-17T23:41:50.107 INFO:teuthology.orchestra.run.smithi038.stderr: 2024-09-17T23:41:50.107 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.56713.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39411.log 2024-09-17T23:41:50.107 INFO:teuthology.orchestra.run.smithi038.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.56713.log.gz 2024-09-17T23:41:50.108 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.59724.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58240.log 2024-09-17T23:41:50.108 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.39411.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35634.log 2024-09-17T23:41:50.111 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.58240.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.59724.log.gz 2024-09-17T23:41:50.111 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64708.log 2024-09-17T23:41:50.111 INFO:teuthology.orchestra.run.smithi038.stderr: 89.7%/var/log/ceph/cluster2-client.admin.35634.log: -- replaced with /var/log/ceph/cluster2-client.mirror.39411.log.gz 2024-09-17T23:41:50.111 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.35634.log.gz 2024-09-17T23:41:50.111 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66415.log 2024-09-17T23:41:50.111 INFO:teuthology.orchestra.run.smithi038.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.58240.log.gz 2024-09-17T23:41:50.113 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.64708.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70474.log 2024-09-17T23:41:50.113 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.66415.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.64708.log.gz 2024-09-17T23:41:50.113 INFO:teuthology.orchestra.run.smithi038.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.66415.log.gz 2024-09-17T23:41:50.113 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54413.log 2024-09-17T23:41:50.113 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60804.log 2024-09-17T23:41:50.114 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.70474.log: /var/log/ceph/cluster2-client.mirror.54413.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.70474.log.gz 2024-09-17T23:41:50.114 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69652.log 2024-09-17T23:41:50.114 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.60804.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.54413.log.gz 2024-09-17T23:41:50.114 INFO:teuthology.orchestra.run.smithi038.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.60804.log.gz 2024-09-17T23:41:50.116 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65650.log 2024-09-17T23:41:50.117 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose/var/log/ceph/cluster1-client.mirror.69652.log: -- /var/log/ceph/cluster2-client.mirror.48335.log 2024-09-17T23:41:50.117 INFO:teuthology.orchestra.run.smithi038.stderr: 87.3%/var/log/ceph/cluster1-client.mirror.65650.log: -- replaced with /var/log/ceph/cluster1-client.mirror.69652.log.gz 2024-09-17T23:41:50.117 INFO:teuthology.orchestra.run.smithi038.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.65650.log.gz 2024-09-17T23:41:50.117 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44394.log 2024-09-17T23:41:50.117 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46454.log 2024-09-17T23:41:50.117 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.44394.log: /var/log/ceph/cluster2-client.mirror.48335.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48335.log.gz 2024-09-17T23:41:50.118 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55194.log 2024-09-17T23:41:50.119 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.46454.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.46454.log.gz 2024-09-17T23:41:50.119 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65830.log 2024-09-17T23:41:50.120 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.55194.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.55194.log.gz 2024-09-17T23:41:50.120 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69314.log 2024-09-17T23:41:50.121 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.65830.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.65830.log.gz 2024-09-17T23:41:50.121 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47637.log 2024-09-17T23:41:50.122 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.69314.log: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.44394.log.gz 2024-09-17T23:41:50.124 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44139.log 2024-09-17T23:41:50.125 INFO:teuthology.orchestra.run.smithi038.stderr: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.69314.log.gz 2024-09-17T23:41:50.125 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.47637.log: 80.2% -- replaced with /var/log/ceph/cluster2-client.mirror.47637.log.gz 2024-09-17T23:41:50.125 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43709.log 2024-09-17T23:41:50.125 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47994.log 2024-09-17T23:41:50.136 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.43709.log: /var/log/ceph/cluster2-client.mirror.44139.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.43709.log.gz 2024-09-17T23:41:50.136 INFO:teuthology.orchestra.run.smithi038.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.47778.log.gz 2024-09-17T23:41:50.136 INFO:teuthology.orchestra.run.smithi038.stderr: 93.4%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52329.log 2024-09-17T23:41:50.136 INFO:teuthology.orchestra.run.smithi038.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.44139.log.gz 2024-09-17T23:41:50.136 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.47994.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44812.log 2024-09-17T23:41:50.136 INFO:teuthology.orchestra.run.smithi038.stderr: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47994.log.gz 2024-09-17T23:41:50.136 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.52329.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62329.log 2024-09-17T23:41:50.136 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose/var/log/ceph/cluster2-client.mirror.44812.log: -- /var/log/ceph/cluster2-client.mirror.67976.log 2024-09-17T23:41:50.137 INFO:teuthology.orchestra.run.smithi038.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.52329.log.gz 2024-09-17T23:41:50.137 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.62329.log: gzip 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.44812.log.gz 2024-09-17T23:41:50.137 INFO:teuthology.orchestra.run.smithi038.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69014.log 2024-09-17T23:41:50.137 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.67976.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65448.log 2024-09-17T23:41:50.137 INFO:teuthology.orchestra.run.smithi038.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.62329.log.gz 2024-09-17T23:41:50.137 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.69014.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62866.log 2024-09-17T23:41:50.137 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.65448.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.69014.log.gz 2024-09-17T23:41:50.137 INFO:teuthology.orchestra.run.smithi038.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.65448.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67023.log 2024-09-17T23:41:50.137 INFO:teuthology.orchestra.run.smithi038.stderr: 2024-09-17T23:41:50.137 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47388.log 2024-09-17T23:41:50.137 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.62866.log: /var/log/ceph/cluster1-client.mirror.67023.log: 91.1% 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67023.log.gz 2024-09-17T23:41:50.137 INFO:teuthology.orchestra.run.smithi038.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.67976.log.gz 2024-09-17T23:41:50.137 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66595.log 2024-09-17T23:41:50.137 INFO:teuthology.orchestra.run.smithi038.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.62866.log.gz 2024-09-17T23:41:50.137 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61745.log 2024-09-17T23:41:50.138 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.47388.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51085.log 2024-09-17T23:41:50.138 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.66595.log: /var/log/ceph/cluster2-client.mirror.61745.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66595.log.gz 2024-09-17T23:41:50.149 INFO:teuthology.orchestra.run.smithi038.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47388.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52893.log 2024-09-17T23:41:50.149 INFO:teuthology.orchestra.run.smithi038.stderr: 2024-09-17T23:41:50.149 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53303.log 2024-09-17T23:41:50.149 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.52893.log: /var/log/ceph/cluster1-client.mirror.51085.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.51085.log.gz 2024-09-17T23:41:50.150 INFO:teuthology.orchestra.run.smithi038.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.52893.log.gz 2024-09-17T23:41:50.150 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.72075.log 2024-09-17T23:41:50.150 INFO:teuthology.orchestra.run.smithi038.stderr: 90.3%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66775.log 2024-09-17T23:41:50.150 INFO:teuthology.orchestra.run.smithi038.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.61745.log.gz/var/log/ceph/cluster2-client.mirror.53303.log: 2024-09-17T23:41:50.150 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.admin.72075.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.72075.log.gz 2024-09-17T23:41:50.150 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53168.log 2024-09-17T23:41:50.150 INFO:teuthology.orchestra.run.smithi038.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.53303.log.gz 2024-09-17T23:41:50.150 INFO:teuthology.orchestra.run.smithi038.stderr:gzip/var/log/ceph/cluster1-client.mirror.66775.log: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60555.log 2024-09-17T23:41:50.150 INFO:teuthology.orchestra.run.smithi038.stderr: 86.9%/var/log/ceph/cluster1-client.mirror.53168.log: -- replaced with /var/log/ceph/cluster1-client.mirror.66775.log.gz 2024-09-17T23:41:50.150 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.53168.log.gz 2024-09-17T23:41:50.150 INFO:teuthology.orchestra.run.smithi038.stderr: /var/log/ceph/cluster2-client.mirror.68758.log 2024-09-17T23:41:50.150 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.60555.log: gzip -5 --verbose -- 87.3% /var/log/ceph/cluster1-client.mirror.61095.log 2024-09-17T23:41:50.150 INFO:teuthology.orchestra.run.smithi038.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.60555.log.gz 2024-09-17T23:41:50.150 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.68758.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68357.log 2024-09-17T23:41:50.151 INFO:teuthology.orchestra.run.smithi038.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.68758.log.gz/var/log/ceph/cluster1-client.mirror.61095.log: 2024-09-17T23:41:50.151 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61475.log 2024-09-17T23:41:50.151 INFO:teuthology.orchestra.run.smithi038.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.61095.log.gz 2024-09-17T23:41:50.151 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose/var/log/ceph/cluster2-client.mirror.68357.log: -- /var/log/ceph/cluster2-client.mirror.56607.log 2024-09-17T23:41:50.151 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.61475.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.68357.log.gz 2024-09-17T23:41:50.151 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56211.log 2024-09-17T23:41:50.151 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.56607.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51419.log 2024-09-17T23:41:50.151 INFO:teuthology.orchestra.run.smithi038.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.61475.log.gz 2024-09-17T23:41:50.151 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.56211.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.56607.log.gz 2024-09-17T23:41:50.151 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40313.log 2024-09-17T23:41:50.151 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.51419.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36182.log 2024-09-17T23:41:50.151 INFO:teuthology.orchestra.run.smithi038.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.56211.log.gz 2024-09-17T23:41:50.151 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.40313.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51873.log 2024-09-17T23:41:50.151 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54736.log 2024-09-17T23:41:50.151 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.51873.log: /var/log/ceph/cluster2-client.admin.36182.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36182.log.gz 2024-09-17T23:41:50.152 INFO:teuthology.orchestra.run.smithi038.stderr: 92.5%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45544.log 2024-09-17T23:41:50.152 INFO:teuthology.orchestra.run.smithi038.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.51419.log.gz 2024-09-17T23:41:50.152 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.40313.log.gz 2024-09-17T23:41:50.152 INFO:teuthology.orchestra.run.smithi038.stderr: --verbose -- /var/log/ceph/cluster2-client.mirror.43838.log 2024-09-17T23:41:50.152 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.54736.log: /var/log/ceph/cluster2-client.mirror.45544.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.54736.log.gz 2024-09-17T23:41:50.152 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45544.log.gz 2024-09-17T23:41:50.152 INFO:teuthology.orchestra.run.smithi038.stderr: --verbose -- /var/log/ceph/cluster2-client.mirror.42542.log 2024-09-17T23:41:50.152 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.43838.log: 91.9% -- replaced with /var/log/ceph/cluster1-client.mirror.51873.log.gz 2024-09-17T23:41:50.152 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55435.log 2024-09-17T23:41:50.152 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.42542.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61340.log 2024-09-17T23:41:50.164 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48785.log 2024-09-17T23:41:50.164 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.55435.log: /var/log/ceph/cluster1-client.mirror.61340.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.55435.log.gz 2024-09-17T23:41:50.164 INFO:teuthology.orchestra.run.smithi038.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.61340.log.gz 2024-09-17T23:41:50.164 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57999.log 2024-09-17T23:41:50.164 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.48785.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58340.log 2024-09-17T23:41:50.164 INFO:teuthology.orchestra.run.smithi038.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48785.log.gz 2024-09-17T23:41:50.164 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.57999.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38787.log 2024-09-17T23:41:50.164 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.58340.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.58340.log.gz 2024-09-17T23:41:50.164 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46393.log 2024-09-17T23:41:50.164 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.admin.38787.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66820.log 2024-09-17T23:41:50.164 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.46393.log: 87.1% -- replaced with /var/log/ceph/cluster1-client.admin.38787.log.gz 2024-09-17T23:41:50.164 INFO:teuthology.orchestra.run.smithi038.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.57999.log.gz 2024-09-17T23:41:50.165 INFO:teuthology.orchestra.run.smithi038.stderr: 93.3% -- replaced with /var/log/ceph/cluster2-client.mirror.43838.log.gz 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46393.log.gz 2024-09-17T23:41:50.165 INFO:teuthology.orchestra.run.smithi038.stderr: 2024-09-17T23:41:50.165 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37403.log 2024-09-17T23:41:50.165 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49186.log/var/log/ceph/cluster1-client.mirror.66820.log: 2024-09-17T23:41:50.165 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.37403.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66820.log.gz 2024-09-17T23:41:50.165 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53341.log 2024-09-17T23:41:50.165 INFO:teuthology.orchestra.run.smithi038.stderr: 84.8% -- replaced with /var/log/ceph/cluster2-client.mirror.37403.log.gz 2024-09-17T23:41:50.165 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52150.log 2024-09-17T23:41:50.165 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.49186.log: gzip -5 --verbose -- 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.49186.log.gz 2024-09-17T23:41:50.165 INFO:teuthology.orchestra.run.smithi038.stderr: /var/log/ceph/cluster2-client.mirror.63799.log 2024-09-17T23:41:50.165 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.52150.log: /var/log/ceph/cluster2-client.mirror.53341.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57535.log 2024-09-17T23:41:50.165 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.63799.log: 88.3% 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.53341.log.gz 2024-09-17T23:41:50.165 INFO:teuthology.orchestra.run.smithi038.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.63799.log.gz 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.52150.log.gz 2024-09-17T23:41:50.165 INFO:teuthology.orchestra.run.smithi038.stderr: 2024-09-17T23:41:50.165 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61589.log 2024-09-17T23:41:50.166 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.57535.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59137.log 2024-09-17T23:41:50.166 INFO:teuthology.orchestra.run.smithi038.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.57535.log.gz 2024-09-17T23:41:50.166 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46635.log 2024-09-17T23:41:50.166 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.59137.log: /var/log/ceph/cluster1-client.mirror.61589.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61589.log.gz 2024-09-17T23:41:50.166 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41224.log 2024-09-17T23:41:50.166 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.46635.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.59137.log.gz 2024-09-17T23:41:50.166 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59406.log 2024-09-17T23:41:50.166 INFO:teuthology.orchestra.run.smithi038.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46635.log.gz 2024-09-17T23:41:50.166 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.admin.41224.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39917.log 2024-09-17T23:41:50.166 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.59406.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59454.log 2024-09-17T23:41:50.166 INFO:teuthology.orchestra.run.smithi038.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.59406.log.gz 2024-09-17T23:41:50.166 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.39917.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39761.log 2024-09-17T23:41:50.166 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.59454.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46335.log 2024-09-17T23:41:50.166 INFO:teuthology.orchestra.run.smithi038.stderr:gzip 87.7% -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45805.log 2024-09-17T23:41:50.167 INFO:teuthology.orchestra.run.smithi038.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.59454.log.gz/var/log/ceph/cluster2-client.mirror.39761.log: 2024-09-17T23:41:50.170 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.46335.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.42542.log.gz 2024-09-17T23:41:50.170 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.41224.log.gz 2024-09-17T23:41:50.171 INFO:teuthology.orchestra.run.smithi038.stderr: -- /var/log/ceph/cluster1-client.mirror.59004.log 2024-09-17T23:41:50.171 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.45805.log: 87.4% 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.39917.log.gz 2024-09-17T23:41:50.171 INFO:teuthology.orchestra.run.smithi038.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.45805.log.gz 2024-09-17T23:41:50.171 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.39761.log.gz 2024-09-17T23:41:50.171 INFO:teuthology.orchestra.run.smithi038.stderr: --verbose -- /var/log/ceph/cluster2-client.mirror.3.69537.log 2024-09-17T23:41:50.171 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55536.log 2024-09-17T23:41:50.171 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.59004.log: /var/log/ceph/cluster2-client.mirror.3.69537.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.59004.log.gz 2024-09-17T23:41:50.171 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60105.log 2024-09-17T23:41:50.171 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.55536.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.55536.log.gz 2024-09-17T23:41:50.171 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63819.log 2024-09-17T23:41:50.171 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.60105.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.60105.log.gz 2024-09-17T23:41:50.171 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69759.log 2024-09-17T23:41:50.172 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.63819.log: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.3.69537.log.gz 2024-09-17T23:41:50.173 INFO:teuthology.orchestra.run.smithi038.stderr: 90.2% -- replaced with /var/log/ceph/cluster2-client.mirror.63819.log.gz 2024-09-17T23:41:50.175 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40429.log 2024-09-17T23:41:50.175 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.69759.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57435.log 2024-09-17T23:41:50.175 INFO:teuthology.orchestra.run.smithi038.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.69759.log.gz 2024-09-17T23:41:50.175 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.40429.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70618.log 2024-09-17T23:41:50.175 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.57435.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39431.log 2024-09-17T23:41:50.176 INFO:teuthology.orchestra.run.smithi038.stderr: 88.5% -- replaced with /var/log/ceph/cluster1-client.mirror.40429.log.gz 2024-09-17T23:41:50.177 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.70618.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.70618.log.gz 2024-09-17T23:41:50.178 INFO:teuthology.orchestra.run.smithi038.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.57435.log.gz 2024-09-17T23:41:50.178 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56520.log 2024-09-17T23:41:50.178 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.39431.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41531.log 2024-09-17T23:41:50.179 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.56520.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.39431.log.gz 2024-09-17T23:41:50.179 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60828.log 2024-09-17T23:41:50.179 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.41531.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.56520.log.gz 2024-09-17T23:41:50.179 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45688.log 2024-09-17T23:41:50.179 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.46731.log 2024-09-17T23:41:50.181 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.60828.log: /var/log/ceph/cluster2-client.mirror.45688.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.60828.log.gz 2024-09-17T23:41:50.181 INFO:teuthology.orchestra.run.smithi038.stderr: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.45688.log.gz 2024-09-17T23:41:50.181 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48318.log 2024-09-17T23:41:50.181 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65695.log 2024-09-17T23:41:50.182 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.2.46731.log: /var/log/ceph/cluster2-client.mirror.48318.log: 80.2% -- replaced with /var/log/ceph/cluster2-client.mirror.48318.log.gz 2024-09-17T23:41:50.183 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58388.log 2024-09-17T23:41:50.183 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.65695.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.65695.log.gz 2024-09-17T23:41:50.183 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54760.log 2024-09-17T23:41:50.186 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.58388.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43456.log 2024-09-17T23:41:50.186 INFO:teuthology.orchestra.run.smithi038.stderr: 91.5% -- replaced with /var/log/ceph/cluster2-client.mirror.2.46731.log.gz 2024-09-17T23:41:50.186 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.54760.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.58388.log.gz 2024-09-17T23:41:50.186 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69880.log 2024-09-17T23:41:50.186 INFO:teuthology.orchestra.run.smithi038.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.54760.log.gz 2024-09-17T23:41:50.186 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50940.log 2024-09-17T23:41:50.186 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.43456.log: /var/log/ceph/cluster1-client.mirror.69880.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62494.log 2024-09-17T23:41:50.186 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.50940.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.50940.log.gz 2024-09-17T23:41:50.186 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48459.log 2024-09-17T23:41:50.187 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.62494.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.62494.log.gz 2024-09-17T23:41:50.188 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57477.log 2024-09-17T23:41:50.190 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.48459.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.43456.log.gz 2024-09-17T23:41:50.192 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60037.log 2024-09-17T23:41:50.192 INFO:teuthology.orchestra.run.smithi038.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.69880.log.gz 2024-09-17T23:41:50.192 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.57477.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58581.log 2024-09-17T23:41:50.196 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.60037.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.57477.log.gz 2024-09-17T23:41:50.196 INFO:teuthology.orchestra.run.smithi038.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.60037.log.gz 2024-09-17T23:41:50.196 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49669.log 2024-09-17T23:41:50.196 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55869.log 2024-09-17T23:41:50.196 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.49669.log: /var/log/ceph/cluster1-client.mirror.58581.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.58581.log.gz 2024-09-17T23:41:50.196 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49669.log.gz 2024-09-17T23:41:50.196 INFO:teuthology.orchestra.run.smithi038.stderr: /var/log/ceph/cluster1-client.mirror.61186.log 2024-09-17T23:41:50.196 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56327.log 2024-09-17T23:41:50.196 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.61186.log: /var/log/ceph/cluster1-client.mirror.55869.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.55869.log.gz 2024-09-17T23:41:50.197 INFO:teuthology.orchestra.run.smithi038.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.61186.log.gz 2024-09-17T23:41:50.197 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42731.log 2024-09-17T23:41:50.198 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45944.log 2024-09-17T23:41:50.198 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.56327.log: /var/log/ceph/cluster2-client.mirror.42731.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.56327.log.gz 2024-09-17T23:41:50.198 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.38916.log 2024-09-17T23:41:50.199 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.45944.log: 90.0% 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.45944.log.gz 2024-09-17T23:41:50.199 INFO:teuthology.orchestra.run.smithi038.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.42731.log.gz 2024-09-17T23:41:50.199 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.36785.log 2024-09-17T23:41:50.199 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57316.log 2024-09-17T23:41:50.199 INFO:teuthology.orchestra.run.smithi038.stderr: 93.3%/var/log/ceph/cluster1-client.mirror.0.38916.log: -- replaced with /var/log/ceph/cluster2-client.mirror.41531.log.gz 2024-09-17T23:41:50.199 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.3.36785.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51398.log 2024-09-17T23:41:50.201 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.57316.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.57316.log.gz 2024-09-17T23:41:50.201 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41878.log 2024-09-17T23:41:50.203 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.51398.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.51398.log.gz 2024-09-17T23:41:50.203 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58980.log 2024-09-17T23:41:50.203 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.41878.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64605.log 2024-09-17T23:41:50.204 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.58980.log: 88.6% -- replaced with /var/log/ceph/cluster1-client.mirror.41878.log.gz 2024-09-17T23:41:50.204 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66168.log 2024-09-17T23:41:50.205 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.64605.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.58980.log.gz 2024-09-17T23:41:50.205 INFO:teuthology.orchestra.run.smithi038.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.64605.log.gz 2024-09-17T23:41:50.213 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40357.log 2024-09-17T23:41:50.214 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.66168.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.66168.log.gz 2024-09-17T23:41:50.214 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49632.log 2024-09-17T23:41:50.215 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.40357.log: 88.3% -- replaced with /var/log/ceph/cluster1-client.mirror.40357.log.gz 2024-09-17T23:41:50.226 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50479.log 2024-09-17T23:41:50.227 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42751.log 2024-09-17T23:41:50.229 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.49632.log: /var/log/ceph/cluster2-client.mirror.50479.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50479.log.gz 2024-09-17T23:41:50.232 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52483.log 2024-09-17T23:41:50.235 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.42751.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.42751.log.gz 2024-09-17T23:41:50.235 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70552.log 2024-09-17T23:41:50.236 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.52483.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.52483.log.gz 2024-09-17T23:41:50.237 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49047.log 2024-09-17T23:41:50.237 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.70552.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.70552.log.gz 2024-09-17T23:41:50.242 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49689.log 2024-09-17T23:41:50.250 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.49047.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68034.log 2024-09-17T23:41:50.261 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40638.log 2024-09-17T23:41:50.262 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.68034.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.68034.log.gz 2024-09-17T23:41:50.263 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.49689.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49689.log.gz 2024-09-17T23:41:50.266 INFO:teuthology.orchestra.run.smithi038.stderr: 93.2% -- replaced with /var/log/ceph/cluster1-client.mirror.0.38916.log.gz 2024-09-17T23:41:50.279 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35242.log 2024-09-17T23:41:50.280 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.admin.40638.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57079.log 2024-09-17T23:41:50.280 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.admin.35242.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.35242.log.gz 2024-09-17T23:41:50.285 INFO:teuthology.orchestra.run.smithi038.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.40638.log.gz 2024-09-17T23:41:50.289 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64088.log 2024-09-17T23:41:50.290 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.57079.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.57079.log.gz 2024-09-17T23:41:50.293 INFO:teuthology.orchestra.run.smithi038.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48966.log.gz 2024-09-17T23:41:50.300 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46255.log 2024-09-17T23:41:50.301 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.64088.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.64088.log.gz 2024-09-17T23:41:50.302 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61565.log 2024-09-17T23:41:50.316 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.46255.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.46415.log.gz 2024-09-17T23:41:50.318 INFO:teuthology.orchestra.run.smithi038.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48622.log.gz 2024-09-17T23:41:50.318 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47932.log 2024-09-17T23:41:50.319 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.61565.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47425.log 2024-09-17T23:41:50.320 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.47932.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.61565.log.gz 2024-09-17T23:41:50.320 INFO:teuthology.orchestra.run.smithi038.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47932.log.gz 2024-09-17T23:41:50.322 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67113.log 2024-09-17T23:41:50.322 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.47425.log: 89.7% -- replaced with /var/log/ceph/cluster1-client.mirror.47425.log.gz 2024-09-17T23:41:50.323 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62659.log 2024-09-17T23:41:50.324 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.67113.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67113.log.gz 2024-09-17T23:41:50.324 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43572.log 2024-09-17T23:41:50.327 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.62659.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.62659.log.gz 2024-09-17T23:41:50.327 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62432.log 2024-09-17T23:41:50.329 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70996.log 2024-09-17T23:41:50.329 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.43572.log: /var/log/ceph/cluster2-client.mirror.62432.log: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.43572.log.gz 2024-09-17T23:41:50.329 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52090.log 2024-09-17T23:41:50.331 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.70996.log: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.70996.log.gz 2024-09-17T23:41:50.332 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69420.log 2024-09-17T23:41:50.333 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.52090.log: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.52090.log.gz 2024-09-17T23:41:50.334 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60603.log 2024-09-17T23:41:50.337 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.69420.log: 91.2% -- replaced with /var/log/ceph/cluster2-client.mirror.69420.log.gz 2024-09-17T23:41:50.337 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51222.log 2024-09-17T23:41:50.338 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.60603.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.60603.log.gz 2024-09-17T23:41:50.339 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.43050.log 2024-09-17T23:41:50.341 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.51222.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.51222.log.gz 2024-09-17T23:41:50.342 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55476.log 2024-09-17T23:41:50.342 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.admin.43050.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.43050.log.gz 2024-09-17T23:41:50.343 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.39798.log 2024-09-17T23:41:50.345 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.55476.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.55476.log.gz 2024-09-17T23:41:50.346 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46740.log 2024-09-17T23:41:50.347 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.2.39798.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42477.log 2024-09-17T23:41:50.347 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.46740.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.46740.log.gz 2024-09-17T23:41:50.349 INFO:teuthology.orchestra.run.smithi038.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.62432.log.gz 2024-09-17T23:41:50.349 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44912.log 2024-09-17T23:41:50.350 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.42477.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44772.log 2024-09-17T23:41:50.352 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.44912.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44912.log.gz 2024-09-17T23:41:50.352 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65403.log 2024-09-17T23:41:50.354 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.44772.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.44772.log.gz 2024-09-17T23:41:50.356 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46134.log 2024-09-17T23:41:50.356 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.65403.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.65403.log.gz 2024-09-17T23:41:50.363 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43113.log 2024-09-17T23:41:50.364 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.46134.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.2.39798.log.gz 2024-09-17T23:41:50.364 INFO:teuthology.orchestra.run.smithi038.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46134.log.gz 2024-09-17T23:41:50.368 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41508.log 2024-09-17T23:41:50.382 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.43113.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.42477.log.gz 2024-09-17T23:41:50.382 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50720.log 2024-09-17T23:41:50.383 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.41508.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52287.log 2024-09-17T23:41:50.385 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.50720.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50720.log.gz 2024-09-17T23:41:50.386 INFO:teuthology.orchestra.run.smithi038.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49896.log.gz 2024-09-17T23:41:50.395 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49127.log 2024-09-17T23:41:50.395 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39961.log 2024-09-17T23:41:50.401 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.49127.log: /var/log/ceph/cluster2-client.mirror.52287.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.52287.log.gz 2024-09-17T23:41:50.401 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60174.log 2024-09-17T23:41:50.402 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.39961.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37517.log 2024-09-17T23:41:50.404 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.60174.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.60174.log.gz 2024-09-17T23:41:50.406 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42038.log 2024-09-17T23:41:50.406 INFO:teuthology.orchestra.run.smithi038.stderr: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.41508.log.gz 2024-09-17T23:41:50.406 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.admin.37517.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45667.log 2024-09-17T23:41:50.406 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.42038.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51061.log 2024-09-17T23:41:50.416 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.45667.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.39961.log.gz 2024-09-17T23:41:50.416 INFO:teuthology.orchestra.run.smithi038.stderr: 80.1% -- replaced with /var/log/ceph/cluster2-client.admin.37517.log.gz 2024-09-17T23:41:50.419 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51499.log 2024-09-17T23:41:50.420 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.51061.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.51061.log.gz 2024-09-17T23:41:50.436 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58716.log 2024-09-17T23:41:50.437 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.51499.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.51499.log.gz 2024-09-17T23:41:50.442 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55001.log 2024-09-17T23:41:50.443 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70217.log 2024-09-17T23:41:50.444 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44832.log 2024-09-17T23:41:50.444 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36785.log 2024-09-17T23:41:50.446 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65763.log 2024-09-17T23:41:50.447 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68316.log 2024-09-17T23:41:50.448 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.58716.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.58716.log.gz 2024-09-17T23:41:50.449 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.55001.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.55001.log.gz 2024-09-17T23:41:50.459 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.70217.log: /var/log/ceph/cluster2-client.mirror.44832.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44832.log.gz 2024-09-17T23:41:50.470 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.36785.log: /var/log/ceph/cluster1-client.mirror.65763.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.65763.log.gz 2024-09-17T23:41:50.477 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66460.log 2024-09-17T23:41:50.477 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.68316.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.68316.log.gz 2024-09-17T23:41:50.490 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64729.log 2024-09-17T23:41:50.490 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.66460.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.66460.log.gz 2024-09-17T23:41:50.502 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65110.log 2024-09-17T23:41:50.502 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.64729.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.64729.log.gz 2024-09-17T23:41:50.512 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64832.log 2024-09-17T23:41:50.512 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.65110.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.65110.log.gz 2024-09-17T23:41:50.515 INFO:teuthology.orchestra.run.smithi038.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.42038.log.gz 2024-09-17T23:41:50.531 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53915.log 2024-09-17T23:41:50.532 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.64832.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.64832.log.gz 2024-09-17T23:41:50.545 INFO:teuthology.orchestra.run.smithi038.stderr: 93.8%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42999.log 2024-09-17T23:41:50.546 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.53915.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.53915.log.gz 2024-09-17T23:41:50.551 INFO:teuthology.orchestra.run.smithi038.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.49632.log.gz 2024-09-17T23:41:50.552 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.32458.log 2024-09-17T23:41:50.564 INFO:teuthology.orchestra.run.smithi006.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-mon.a.log.gz 2024-09-17T23:41:50.566 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.42999.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70909.log 2024-09-17T23:41:50.566 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.admin.32458.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.32458.log.gz 2024-09-17T23:41:50.578 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46533.log 2024-09-17T23:41:50.579 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.70909.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.70909.log.gz 2024-09-17T23:41:50.587 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60648.log 2024-09-17T23:41:50.591 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.46533.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46533.log.gz 2024-09-17T23:41:50.601 INFO:teuthology.orchestra.run.smithi038.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.43113.log.gz 2024-09-17T23:41:50.602 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67826.log 2024-09-17T23:41:50.603 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.60648.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.60648.log.gz 2024-09-17T23:41:50.604 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50410.log 2024-09-17T23:41:50.604 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.67826.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.67826.log.gz 2024-09-17T23:41:50.606 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60057.log 2024-09-17T23:41:50.606 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.50410.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.50410.log.gz 2024-09-17T23:41:50.607 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54929.log 2024-09-17T23:41:50.607 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.60057.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.60057.log.gz 2024-09-17T23:41:50.608 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46960.log 2024-09-17T23:41:50.608 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.54929.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.54929.log.gz 2024-09-17T23:41:50.608 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64171.log 2024-09-17T23:41:50.613 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.46960.log: 84.6% -- replaced with /var/log/ceph/cluster2-client.mirror.46960.log.gz 2024-09-17T23:41:50.614 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69379.log 2024-09-17T23:41:50.614 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.64171.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.64171.log.gz 2024-09-17T23:41:50.616 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41068.log 2024-09-17T23:41:50.617 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.69379.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.69379.log.gz 2024-09-17T23:41:50.617 INFO:teuthology.orchestra.run.smithi038.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49047.log.gz 2024-09-17T23:41:50.617 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63261.log 2024-09-17T23:41:50.617 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.41068.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48011.log 2024-09-17T23:41:50.619 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.63261.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.63261.log.gz 2024-09-17T23:41:50.620 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54391.log 2024-09-17T23:41:50.623 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.48011.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.41068.log.gz 2024-09-17T23:41:50.623 INFO:teuthology.orchestra.run.smithi038.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48011.log.gz 2024-09-17T23:41:50.623 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.71781.log 2024-09-17T23:41:50.625 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.54391.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.54391.log.gz 2024-09-17T23:41:50.626 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.72117.log 2024-09-17T23:41:50.627 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.admin.71781.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.71781.log.gz 2024-09-17T23:41:50.627 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57195.log 2024-09-17T23:41:50.631 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.admin.72117.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.72117.log.gz 2024-09-17T23:41:50.632 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63117.log 2024-09-17T23:41:50.633 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.57195.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.57195.log.gz 2024-09-17T23:41:50.633 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48944.log 2024-09-17T23:41:50.634 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.63117.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.63117.log.gz 2024-09-17T23:41:50.635 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44566.log 2024-09-17T23:41:50.636 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.48944.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48944.log.gz 2024-09-17T23:41:50.636 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56348.log 2024-09-17T23:41:50.649 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.44566.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.44566.log.gz 2024-09-17T23:41:50.650 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49208.log 2024-09-17T23:41:50.652 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.56348.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.56348.log.gz 2024-09-17T23:41:50.653 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51656.log 2024-09-17T23:41:50.667 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.49208.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45262.log 2024-09-17T23:41:50.669 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.51656.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.51656.log.gz 2024-09-17T23:41:50.687 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52025.log 2024-09-17T23:41:50.705 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.45262.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68566.log 2024-09-17T23:41:50.705 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.52025.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.52025.log.gz 2024-09-17T23:41:50.724 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45220.log 2024-09-17T23:41:50.725 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.68566.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.68566.log.gz 2024-09-17T23:41:50.738 INFO:teuthology.orchestra.run.smithi038.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.46335.log.gz 2024-09-17T23:41:50.740 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62057.log 2024-09-17T23:41:50.742 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.45220.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.45220.log.gz 2024-09-17T23:41:50.750 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55122.log 2024-09-17T23:41:50.759 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.62057.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57414.log 2024-09-17T23:41:50.768 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.55122.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36602.log 2024-09-17T23:41:50.772 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.57414.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.55122.log.gz 2024-09-17T23:41:50.775 INFO:teuthology.orchestra.run.smithi038.stderr: 90.2% -- replaced with /var/log/ceph/cluster2-client.mirror.62057.log.gz 2024-09-17T23:41:50.781 INFO:teuthology.orchestra.run.smithi038.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.57414.log.gz 2024-09-17T23:41:50.788 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70409.log 2024-09-17T23:41:50.788 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.admin.36602.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36602.log.gz 2024-09-17T23:41:50.797 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60736.log 2024-09-17T23:41:50.805 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.70409.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67914.log 2024-09-17T23:41:50.805 INFO:teuthology.orchestra.run.smithi038.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.70409.log.gz 2024-09-17T23:41:50.812 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.60736.log: 89.8%gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39735.log 2024-09-17T23:41:50.812 INFO:teuthology.orchestra.run.smithi038.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.60736.log.gz 2024-09-17T23:41:50.819 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.67914.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47130.log 2024-09-17T23:41:50.819 INFO:teuthology.orchestra.run.smithi038.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.67914.log.gz 2024-09-17T23:41:50.825 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.admin.39735.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54173.log 2024-09-17T23:41:50.832 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41902.log 2024-09-17T23:41:50.834 INFO:teuthology.orchestra.run.smithi038.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.45667.log.gz 2024-09-17T23:41:50.836 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51302.log 2024-09-17T23:41:50.836 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.41902.log: /var/log/ceph/cluster1-client.mirror.54173.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40057.log 2024-09-17T23:41:50.837 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.51302.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.51302.log.gz 2024-09-17T23:41:50.837 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-osd.2.log 2024-09-17T23:41:50.838 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.40057.log: /var/log/ceph/cluster2-client.mirror.47130.log: 80.2% -- replaced with /var/log/ceph/cluster2-client.mirror.47130.log.gz 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.54173.log.gz 2024-09-17T23:41:50.838 INFO:teuthology.orchestra.run.smithi038.stderr: 2024-09-17T23:41:50.838 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68879.log 2024-09-17T23:41:50.841 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53626.log 2024-09-17T23:41:50.841 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.68879.log: /var/log/ceph/cluster2-osd.2.log: 88.7% -- replaced with /var/log/ceph/cluster1-client.mirror.41902.log.gz 2024-09-17T23:41:50.841 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48924.log 2024-09-17T23:41:50.842 INFO:teuthology.orchestra.run.smithi038.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.40057.log.gz 2024-09-17T23:41:50.842 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.53626.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.53626.log.gz 2024-09-17T23:41:50.842 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35886.log 2024-09-17T23:41:50.843 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.48924.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48924.log.gz 2024-09-17T23:41:50.843 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35758.log 2024-09-17T23:41:50.845 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.admin.35886.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.35886.log.gz 2024-09-17T23:41:50.845 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.67448.log 2024-09-17T23:41:50.845 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.admin.35758.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.35758.log.gz 2024-09-17T23:41:50.846 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35866.log 2024-09-17T23:41:50.846 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.1.67448.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.39735.log.gz 2024-09-17T23:41:50.859 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63613.log 2024-09-17T23:41:50.860 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.admin.35866.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.35866.log.gz 2024-09-17T23:41:50.862 INFO:teuthology.orchestra.run.smithi038.stderr: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.1.67448.log.gz 2024-09-17T23:41:50.868 INFO:teuthology.orchestra.run.smithi038.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.70217.log.gz 2024-09-17T23:41:50.869 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50241.log 2024-09-17T23:41:50.869 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.63613.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.63613.log.gz 2024-09-17T23:41:50.871 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49711.log 2024-09-17T23:41:50.871 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.50241.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.50241.log.gz 2024-09-17T23:41:50.871 INFO:teuthology.orchestra.run.smithi038.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48459.log.gz 2024-09-17T23:41:50.885 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55102.log 2024-09-17T23:41:50.900 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.49711.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58956.log 2024-09-17T23:41:50.901 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.55102.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.55102.log.gz 2024-09-17T23:41:50.925 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46862.log 2024-09-17T23:41:50.925 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.58956.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.58956.log.gz 2024-09-17T23:41:50.938 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48214.log 2024-09-17T23:41:50.938 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.46862.log: 80.2% -- replaced with /var/log/ceph/cluster2-client.mirror.46862.log.gz 2024-09-17T23:41:50.947 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66483.log 2024-09-17T23:41:50.957 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.48214.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.46255.log.gz 2024-09-17T23:41:50.961 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39043.log 2024-09-17T23:41:50.961 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.66483.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66483.log.gz 2024-09-17T23:41:50.975 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44892.log 2024-09-17T23:41:50.982 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.39043.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.39043.log.gz 2024-09-17T23:41:50.988 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68394.log 2024-09-17T23:41:50.989 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.44892.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44892.log.gz 2024-09-17T23:41:51.000 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64274.log 2024-09-17T23:41:51.000 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.68394.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.68394.log.gz 2024-09-17T23:41:51.016 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45367.log 2024-09-17T23:41:51.017 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.64274.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.64274.log.gz 2024-09-17T23:41:51.036 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41950.log 2024-09-17T23:41:51.037 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.45367.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.45367.log.gz 2024-09-17T23:41:51.039 INFO:teuthology.orchestra.run.smithi038.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.42999.log.gz 2024-09-17T23:41:51.040 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61227.log 2024-09-17T23:41:51.041 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.41950.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64419.log 2024-09-17T23:41:51.041 INFO:teuthology.orchestra.run.smithi038.stderr: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.41950.log.gz 2024-09-17T23:41:51.044 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.61227.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.61227.log.gz 2024-09-17T23:41:51.044 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.71907.log 2024-09-17T23:41:51.046 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.64419.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.64419.log.gz 2024-09-17T23:41:51.046 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71219.log 2024-09-17T23:41:51.046 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.admin.71907.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.71907.log.gz 2024-09-17T23:41:51.047 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55387.log 2024-09-17T23:41:51.047 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.71219.log: 88.6% -- replaced with /var/log/ceph/cluster1-client.mirror.71219.log.gz 2024-09-17T23:41:51.049 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45608.log 2024-09-17T23:41:51.049 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.55387.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.55387.log.gz 2024-09-17T23:41:51.050 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68857.log 2024-09-17T23:41:51.050 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.45608.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45608.log.gz 2024-09-17T23:41:51.052 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52546.log 2024-09-17T23:41:51.052 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.68857.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.68857.log.gz 2024-09-17T23:41:51.052 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44692.log 2024-09-17T23:41:51.053 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.52546.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.52546.log.gz 2024-09-17T23:41:51.054 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43348.log 2024-09-17T23:41:51.056 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.44692.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.44692.log.gz 2024-09-17T23:41:51.056 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68587.log 2024-09-17T23:41:51.057 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.43348.log: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.43348.log.gz 2024-09-17T23:41:51.057 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55773.log 2024-09-17T23:41:51.059 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.68587.log: 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.68587.log.gz 2024-09-17T23:41:51.060 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.71655.log 2024-09-17T23:41:51.061 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.55773.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.55773.log.gz 2024-09-17T23:41:51.061 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62515.log 2024-09-17T23:41:51.062 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.admin.71655.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.71655.log.gz 2024-09-17T23:41:51.062 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42104.log 2024-09-17T23:41:51.064 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.62515.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.62515.log.gz 2024-09-17T23:41:51.064 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63551.log 2024-09-17T23:41:51.064 INFO:teuthology.orchestra.run.smithi038.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.3.36785.log.gz 2024-09-17T23:41:51.064 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.42104.log: 93.8%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44609.log 2024-09-17T23:41:51.065 INFO:teuthology.orchestra.run.smithi038.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.49127.log.gz 2024-09-17T23:41:51.065 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.63551.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51760.log 2024-09-17T23:41:51.068 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose/var/log/ceph/cluster1-client.mirror.44609.log: -- /var/log/ceph/cluster1-client.mirror.57732.log 2024-09-17T23:41:51.068 INFO:teuthology.orchestra.run.smithi038.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.63551.log.gz 2024-09-17T23:41:51.069 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58123.log 2024-09-17T23:41:51.069 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.57732.log: /var/log/ceph/cluster1-client.mirror.51760.log: 83.5% -- replaced with /var/log/ceph/cluster1-client.mirror.57732.log.gz 2024-09-17T23:41:51.069 INFO:teuthology.orchestra.run.smithi038.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.51760.log.gz 2024-09-17T23:41:51.069 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40962.log 2024-09-17T23:41:51.069 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.58123.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48053.log 2024-09-17T23:41:51.069 INFO:teuthology.orchestra.run.smithi038.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.58123.log.gz 2024-09-17T23:41:51.069 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.40962.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66798.log 2024-09-17T23:41:51.069 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose --/var/log/ceph/cluster2-client.mirror.48053.log: /var/log/ceph/cluster2-client.mirror.3.38738.log 2024-09-17T23:41:51.070 INFO:teuthology.orchestra.run.smithi038.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.40962.log.gz 2024-09-17T23:41:51.070 INFO:teuthology.orchestra.run.smithi038.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.44609.log.gz 2024-09-17T23:41:51.071 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.66798.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66798.log.gz 2024-09-17T23:41:51.077 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42655.log 2024-09-17T23:41:51.078 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.3.38738.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53578.log 2024-09-17T23:41:51.078 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.admin.42655.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.admin.42655.log.gz 2024-09-17T23:41:51.079 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41371.log 2024-09-17T23:41:51.082 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.53578.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47066.log 2024-09-17T23:41:51.083 INFO:teuthology.orchestra.run.smithi038.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.53578.log.gz 2024-09-17T23:41:51.083 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.41371.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52652.log 2024-09-17T23:41:51.083 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.47066.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68837.log 2024-09-17T23:41:51.084 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.52652.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47066.log.gz 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.52652.log.gz 2024-09-17T23:41:51.084 INFO:teuthology.orchestra.run.smithi038.stderr: 2024-09-17T23:41:51.084 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66370.log 2024-09-17T23:41:51.084 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose --/var/log/ceph/cluster2-client.mirror.68837.log: /var/log/ceph/cluster2-client.mirror.64646.log 2024-09-17T23:41:51.085 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.66370.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66370.log.gz 2024-09-17T23:41:51.088 INFO:teuthology.orchestra.run.smithi038.stderr: 88.9% -- replaced with /var/log/ceph/cluster1-client.mirror.41371.log.gz 2024-09-17T23:41:51.090 INFO:teuthology.orchestra.run.smithi038.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.68837.log.gz 2024-09-17T23:41:51.090 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42082.log 2024-09-17T23:41:51.091 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51133.log 2024-09-17T23:41:51.091 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.42082.log: /var/log/ceph/cluster2-client.mirror.64646.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69817.log 2024-09-17T23:41:51.093 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.51133.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.64646.log.gz 2024-09-17T23:41:51.094 INFO:teuthology.orchestra.run.smithi038.stderr:gzip 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.51133.log.gz 2024-09-17T23:41:51.094 INFO:teuthology.orchestra.run.smithi038.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53433.log 2024-09-17T23:41:51.094 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37328.log 2024-09-17T23:41:51.094 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.53433.log: /var/log/ceph/cluster2-client.mirror.69817.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.53433.log.gz 2024-09-17T23:41:51.094 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61912.log 2024-09-17T23:41:51.094 INFO:teuthology.orchestra.run.smithi038.stderr: 89.5%/var/log/ceph/cluster1-client.mirror.37328.log: -- replaced with /var/log/ceph/cluster2-client.mirror.69817.log.gz 2024-09-17T23:41:51.095 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.61912.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.61912.log.gz 2024-09-17T23:41:51.095 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66910.log 2024-09-17T23:41:51.097 INFO:teuthology.orchestra.run.smithi038.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.42082.log.gz 2024-09-17T23:41:51.097 INFO:teuthology.orchestra.run.smithi038.stderr: 53.4% -- replaced with /var/log/ceph/cluster1-client.mirror.37328.log.gz 2024-09-17T23:41:51.097 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39635.log 2024-09-17T23:41:51.103 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.66910.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.66910.log.gz 2024-09-17T23:41:51.103 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66280.log 2024-09-17T23:41:51.103 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67956.log 2024-09-17T23:41:51.103 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.66280.log: /var/log/ceph/cluster1-client.mirror.39635.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.66280.log.gz 2024-09-17T23:41:51.103 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67045.log 2024-09-17T23:41:51.103 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66843.log 2024-09-17T23:41:51.103 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.67045.log: /var/log/ceph/cluster2-client.mirror.67956.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67045.log.gz 2024-09-17T23:41:51.103 INFO:teuthology.orchestra.run.smithi038.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.67956.log.gz 2024-09-17T23:41:51.103 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59631.log 2024-09-17T23:41:51.103 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.66843.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55990.log 2024-09-17T23:41:51.103 INFO:teuthology.orchestra.run.smithi038.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66843.log.gz 2024-09-17T23:41:51.103 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.59631.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.58843.log 2024-09-17T23:41:51.104 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.55990.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.59631.log.gz 2024-09-17T23:41:51.104 INFO:teuthology.orchestra.run.smithi038.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.55990.log.gz 2024-09-17T23:41:51.104 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63199.log 2024-09-17T23:41:51.106 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.0.58843.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58079.log 2024-09-17T23:41:51.108 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.63199.log: 93.2% -- replaced with /var/log/ceph/cluster1-client.mirror.0.58843.log.gz 2024-09-17T23:41:51.111 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49977.log 2024-09-17T23:41:51.111 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.58079.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.63199.log.gz 2024-09-17T23:41:51.111 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38738.log 2024-09-17T23:41:51.111 INFO:teuthology.orchestra.run.smithi038.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.58079.log.gz 2024-09-17T23:41:51.111 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.49977.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39679.log 2024-09-17T23:41:51.117 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.38738.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37277.log 2024-09-17T23:41:51.132 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.39679.log: 95.3% -- replaced with /var/log/ceph/cluster1-client.mirror.38738.log.gz 2024-09-17T23:41:51.132 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52763.log 2024-09-17T23:41:51.148 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.37277.log: 93.3% 2.3% -- replaced with /var/log/ceph/cluster1-client.mirror.37277.log.gz 2024-09-17T23:41:51.148 INFO:teuthology.orchestra.run.smithi038.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.39635.log.gz 2024-09-17T23:41:51.150 INFO:teuthology.orchestra.run.smithi038.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49208.log.gz 2024-09-17T23:41:51.150 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36756.log 2024-09-17T23:41:51.150 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37209.log 2024-09-17T23:41:51.150 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.52763.log: /var/log/ceph/cluster2-client.mirror.36756.log: 89.7%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68799.log 2024-09-17T23:41:51.150 INFO:teuthology.orchestra.run.smithi038.stderr: 90.8% -- replaced with /var/log/ceph/cluster2-client.mirror.36756.log.gz 2024-09-17T23:41:51.152 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.admin.37209.log: 79.9% -- replaced with /var/log/ceph/cluster1-client.admin.37209.log.gz 2024-09-17T23:41:51.152 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70831.log 2024-09-17T23:41:51.157 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.68799.log: 91.1% -- replaced with /var/log/ceph/cluster2-client.mirror.68799.log.gz 2024-09-17T23:41:51.157 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67891.log 2024-09-17T23:41:51.158 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.70831.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.70831.log.gz 2024-09-17T23:41:51.158 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66933.log 2024-09-17T23:41:51.160 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.67891.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67891.log.gz 2024-09-17T23:41:51.160 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40664.log 2024-09-17T23:41:51.161 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.66933.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.66933.log.gz 2024-09-17T23:41:51.161 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44632.log 2024-09-17T23:41:51.162 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.40664.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40333.log 2024-09-17T23:41:51.164 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.44632.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.44632.log.gz 2024-09-17T23:41:51.164 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58694.log 2024-09-17T23:41:51.165 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.40333.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.71480.log 2024-09-17T23:41:51.166 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.58694.log: -- replaced with /var/log/ceph/cluster2-client.mirror.52763.log.gz 2024-09-17T23:41:51.167 INFO:teuthology.orchestra.run.smithi038.stderr: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.58694.log.gz 2024-09-17T23:41:51.167 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65290.log 2024-09-17T23:41:51.167 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.admin.71480.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.71480.log.gz 2024-09-17T23:41:51.168 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67448.log 2024-09-17T23:41:51.168 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.65290.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.65290.log.gz 2024-09-17T23:41:51.169 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57882.log 2024-09-17T23:41:51.170 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.67448.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.67448.log.gz 2024-09-17T23:41:51.170 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50087.log 2024-09-17T23:41:51.171 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.57882.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.57882.log.gz 2024-09-17T23:41:51.171 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63282.log 2024-09-17T23:41:51.172 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.50087.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.40333.log.gz 2024-09-17T23:41:51.172 INFO:teuthology.orchestra.run.smithi038.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50087.log.gz 2024-09-17T23:41:51.173 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.3.36894.log 2024-09-17T23:41:51.174 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.63282.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.63282.log.gz 2024-09-17T23:41:51.175 INFO:teuthology.orchestra.run.smithi038.stderr: 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.40664.log.gz 2024-09-17T23:41:51.176 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67000.log 2024-09-17T23:41:51.177 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.3.36894.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46899.log 2024-09-17T23:41:51.177 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.67000.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.67000.log.gz 2024-09-17T23:41:51.178 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.71991.log 2024-09-17T23:41:51.179 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.46899.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46899.log.gz 2024-09-17T23:41:51.181 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69694.log 2024-09-17T23:41:51.181 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.admin.71991.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.71991.log.gz 2024-09-17T23:41:51.181 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56665.log 2024-09-17T23:41:51.182 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.69694.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.69694.log.gz 2024-09-17T23:41:51.182 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66033.log 2024-09-17T23:41:51.183 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.56665.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.56665.log.gz 2024-09-17T23:41:51.184 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69800.log 2024-09-17T23:41:51.185 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.66033.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.66033.log.gz 2024-09-17T23:41:51.185 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57838.log 2024-09-17T23:41:51.185 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.69800.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.69800.log.gz 2024-09-17T23:41:51.186 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59478.log 2024-09-17T23:41:51.188 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.57838.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.57838.log.gz 2024-09-17T23:41:51.188 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40559.log 2024-09-17T23:41:51.189 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.59478.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.59478.log.gz 2024-09-17T23:41:51.190 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44182.log 2024-09-17T23:41:51.190 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.40559.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49531.log 2024-09-17T23:41:51.197 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.44182.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.44182.log.gz 2024-09-17T23:41:51.197 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56782.log 2024-09-17T23:41:51.199 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.49531.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.40559.log.gz 2024-09-17T23:41:51.199 INFO:teuthology.orchestra.run.smithi038.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.49531.log.gz 2024-09-17T23:41:51.200 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71130.log 2024-09-17T23:41:51.202 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.56782.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.56782.log.gz 2024-09-17T23:41:51.202 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55281.log 2024-09-17T23:41:51.203 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.71130.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.71130.log.gz 2024-09-17T23:41:51.204 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55498.log 2024-09-17T23:41:51.204 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.55281.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.55281.log.gz 2024-09-17T23:41:51.205 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66235.log 2024-09-17T23:41:51.207 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.55498.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.55498.log.gz 2024-09-17T23:41:51.207 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47167.log 2024-09-17T23:41:51.208 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.66235.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.66235.log.gz 2024-09-17T23:41:51.208 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36224.log 2024-09-17T23:41:51.210 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.47167.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47167.log.gz 2024-09-17T23:41:51.210 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38307.log 2024-09-17T23:41:51.211 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.admin.36224.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36224.log.gz 2024-09-17T23:41:51.211 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42562.log 2024-09-17T23:41:51.212 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.38307.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64667.log 2024-09-17T23:41:51.213 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.42562.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68926.log 2024-09-17T23:41:51.214 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.64667.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.64667.log.gz 2024-09-17T23:41:51.215 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54016.log 2024-09-17T23:41:51.218 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.68926.log: 87.3%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46054.log 2024-09-17T23:41:51.218 INFO:teuthology.orchestra.run.smithi038.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.68926.log.gz 2024-09-17T23:41:51.220 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.54016.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.54016.log.gz 2024-09-17T23:41:51.220 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56862.log 2024-09-17T23:41:51.222 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.46054.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.46054.log.gz 2024-09-17T23:41:51.222 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53978.log 2024-09-17T23:41:51.224 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.56862.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.56862.log.gz 2024-09-17T23:41:51.224 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45140.log 2024-09-17T23:41:51.225 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.53978.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.53978.log.gz 2024-09-17T23:41:51.225 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54471.log 2024-09-17T23:41:51.227 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.45140.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.45140.log.gz 2024-09-17T23:41:51.227 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60893.log 2024-09-17T23:41:51.229 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.54471.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.54471.log.gz 2024-09-17T23:41:51.229 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64398.log 2024-09-17T23:41:51.231 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.60893.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.60893.log.gz 2024-09-17T23:41:51.232 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46176.log 2024-09-17T23:41:51.237 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.64398.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.64398.log.gz 2024-09-17T23:41:51.238 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44480.log 2024-09-17T23:41:51.239 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.46176.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.42562.log.gz 2024-09-17T23:41:51.251 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64997.log 2024-09-17T23:41:51.255 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.44480.log: 91.0% -- replaced with /var/log/ceph/cluster2-client.mirror.3.36894.log.gz 2024-09-17T23:41:51.261 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.45944.log 2024-09-17T23:41:51.262 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.64997.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.64997.log.gz 2024-09-17T23:41:51.270 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49573.log 2024-09-17T23:41:51.273 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.0.45944.log: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.44480.log.gz 2024-09-17T23:41:51.287 INFO:teuthology.orchestra.run.smithi038.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.0.45944.log.gz 2024-09-17T23:41:51.289 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55797.log 2024-09-17T23:41:51.289 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.49573.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.49573.log.gz 2024-09-17T23:41:51.298 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56565.log 2024-09-17T23:41:51.298 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.55797.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.55797.log.gz 2024-09-17T23:41:51.309 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63055.log 2024-09-17T23:41:51.311 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.56565.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.56565.log.gz 2024-09-17T23:41:51.318 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.27622.log 2024-09-17T23:41:51.318 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.63055.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.63055.log.gz 2024-09-17T23:41:51.328 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35759.log 2024-09-17T23:41:51.340 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph.tmp-client.admin.27622.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.27622.log.gz 2024-09-17T23:41:51.346 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64853.log 2024-09-17T23:41:51.346 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.admin.35759.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.35759.log.gz 2024-09-17T23:41:51.363 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48844.log 2024-09-17T23:41:51.364 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.64853.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.64853.log.gz 2024-09-17T23:41:51.372 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59832.log 2024-09-17T23:41:51.381 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.48844.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44460.log 2024-09-17T23:41:51.385 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.59832.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48844.log.gz 2024-09-17T23:41:51.390 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45182.log 2024-09-17T23:41:51.394 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.44460.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.59832.log.gz 2024-09-17T23:41:51.408 INFO:teuthology.orchestra.run.smithi038.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.44460.log.gz 2024-09-17T23:41:51.410 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36893.log 2024-09-17T23:41:51.420 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.45182.log: gzip -5 --verbose 94.1% -- replaced with /var/log/ceph/cluster2-client.mirror.3.38738.log.gz 2024-09-17T23:41:51.420 INFO:teuthology.orchestra.run.smithi038.stderr: -- /var/log/ceph/cluster1-client.mirror.59856.log 2024-09-17T23:41:51.420 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.36893.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48192.log 2024-09-17T23:41:51.422 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.59856.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.59856.log.gz 2024-09-17T23:41:51.422 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38654.log 2024-09-17T23:41:51.422 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.48192.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48192.log.gz 2024-09-17T23:41:51.423 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44417.log 2024-09-17T23:41:51.424 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.admin.38654.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63075.log 2024-09-17T23:41:51.425 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.44417.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.38654.log.gz 2024-09-17T23:41:51.430 INFO:teuthology.orchestra.run.smithi038.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48214.log.gz 93.3% -- replaced with /var/log/ceph/cluster2-client.mirror.44417.log.gz 2024-09-17T23:41:51.430 INFO:teuthology.orchestra.run.smithi038.stderr: 2024-09-17T23:41:51.431 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50386.log 2024-09-17T23:41:51.432 INFO:teuthology.orchestra.run.smithi038.stderr: 73.6% -- replaced with /var/log/ceph/cluster1-client.mirror.36893.log.gz 2024-09-17T23:41:51.435 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.63075.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.63075.log.gz 2024-09-17T23:41:51.448 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39937.log 2024-09-17T23:41:51.449 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.50386.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.50386.log.gz 2024-09-17T23:41:51.450 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61995.log 2024-09-17T23:41:51.451 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.39937.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48580.log 2024-09-17T23:41:51.454 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.61995.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.39937.log.gz 2024-09-17T23:41:51.454 INFO:teuthology.orchestra.run.smithi038.stderr: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.61995.log.gz 2024-09-17T23:41:51.457 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66618.log 2024-09-17T23:41:51.458 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.48580.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48580.log.gz 2024-09-17T23:41:51.459 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51282.log 2024-09-17T23:41:51.460 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.66618.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.66618.log.gz 2024-09-17T23:41:51.460 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39615.log 2024-09-17T23:41:51.461 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.51282.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.51282.log.gz 2024-09-17T23:41:51.462 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52194.log 2024-09-17T23:41:51.463 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.39615.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63571.log 2024-09-17T23:41:51.463 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.52194.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.52194.log.gz 2024-09-17T23:41:51.464 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63137.log 2024-09-17T23:41:51.468 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.63571.log: 90.2% -- replaced with /var/log/ceph/cluster2-client.mirror.63571.log.gz 2024-09-17T23:41:51.468 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48542.log 2024-09-17T23:41:51.471 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.63137.log: 90.2% -- replaced with /var/log/ceph/cluster2-client.mirror.63137.log.gz 2024-09-17T23:41:51.471 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39527.log 2024-09-17T23:41:51.493 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.48542.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56231.log 2024-09-17T23:41:51.502 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.39527.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.39615.log.gz 2024-09-17T23:41:51.507 INFO:teuthology.orchestra.run.smithi038.stderr: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.39527.log.gz 2024-09-17T23:41:51.508 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.56231.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.56231.log.gz 2024-09-17T23:41:51.513 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40405.log 2024-09-17T23:41:51.529 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50458.log 2024-09-17T23:41:51.536 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.40405.log: gzip -5 --verbose -- /var/log/ceph/cluster2.audit.log 2024-09-17T23:41:51.536 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.50458.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.40405.log.gz 2024-09-17T23:41:51.547 INFO:teuthology.orchestra.run.smithi038.stderr: 87.6%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40938.log 2024-09-17T23:41:51.547 INFO:teuthology.orchestra.run.smithi038.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.50458.log.gz 2024-09-17T23:41:51.556 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2.audit.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55239.log 2024-09-17T23:41:51.563 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.40938.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43372.log 2024-09-17T23:41:51.570 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.55239.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61704.log 2024-09-17T23:41:51.571 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.43372.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.55239.log.gz 2024-09-17T23:41:51.582 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65560.log 2024-09-17T23:41:51.583 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.61704.log: 89.2% 88.3% -- replaced with /var/log/ceph/cluster1-client.mirror.43372.log.gz 2024-09-17T23:41:51.584 INFO:teuthology.orchestra.run.smithi038.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49711.log.gz 2024-09-17T23:41:51.584 INFO:teuthology.orchestra.run.smithi038.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.61704.log.gz 2024-09-17T23:41:51.585 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68543.log 2024-09-17T23:41:51.585 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.65560.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.65560.log.gz 2024-09-17T23:41:51.586 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54302.log 2024-09-17T23:41:51.586 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.68543.log: 87.1% -- replaced with /var/log/ceph/cluster1-client.mirror.68543.log.gz 2024-09-17T23:41:51.587 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43729.log 2024-09-17T23:41:51.588 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.54302.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.54302.log.gz 2024-09-17T23:41:51.588 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.36697.log 2024-09-17T23:41:51.594 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.43729.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.43729.log.gz 2024-09-17T23:41:51.595 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61364.log 2024-09-17T23:41:51.596 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.0.36697.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50961.log 2024-09-17T23:41:51.597 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.61364.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.61364.log.gz 2024-09-17T23:41:51.598 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44096.log 2024-09-17T23:41:51.600 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.50961.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50961.log.gz 2024-09-17T23:41:51.600 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42357.log 2024-09-17T23:41:51.607 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.44096.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.44096.log.gz 2024-09-17T23:41:51.607 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.71739.log 2024-09-17T23:41:51.608 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.42357.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58282.log 2024-09-17T23:41:51.608 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.admin.71739.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.71739.log.gz 2024-09-17T23:41:51.611 INFO:teuthology.orchestra.run.smithi038.stderr: 94.3% -- replaced with /var/log/ceph/cluster2-client.mirror.36785.log.gz 2024-09-17T23:41:51.612 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66888.log 2024-09-17T23:41:51.613 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.58282.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.58282.log.gz 2024-09-17T23:41:51.614 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35760.log 2024-09-17T23:41:51.615 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.66888.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.66888.log.gz 2024-09-17T23:41:51.615 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63964.log 2024-09-17T23:41:51.616 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.admin.35760.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.35760.log.gz 2024-09-17T23:41:51.617 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56062.log 2024-09-17T23:41:51.618 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.63964.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.63964.log.gz 2024-09-17T23:41:51.618 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58320.log 2024-09-17T23:41:51.619 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.56062.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.56062.log.gz 2024-09-17T23:41:51.620 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60961.log 2024-09-17T23:41:51.621 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.58320.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.58320.log.gz 2024-09-17T23:41:51.621 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46615.log 2024-09-17T23:41:51.623 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.60961.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.60961.log.gz 2024-09-17T23:41:51.623 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66010.log 2024-09-17T23:41:51.625 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.46615.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.46615.log.gz 2024-09-17T23:41:51.625 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63737.log 2024-09-17T23:41:51.626 INFO:teuthology.orchestra.run.smithi038.stderr: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.0.36697.log.gz 2024-09-17T23:41:51.627 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.66010.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66010.log.gz 2024-09-17T23:41:51.627 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57123.log 2024-09-17T23:41:51.627 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.63737.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.63737.log.gz 2024-09-17T23:41:51.628 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60511.log 2024-09-17T23:41:51.630 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.57123.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.57123.log.gz 2024-09-17T23:41:51.630 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.68879.log.gz 2024-09-17T23:41:51.630 INFO:teuthology.orchestra.run.smithi038.stderr: /var/log/ceph/cluster2-client.mirror.53261.log 2024-09-17T23:41:51.630 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57171.log 2024-09-17T23:41:51.631 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.60511.log: /var/log/ceph/cluster2-client.mirror.53261.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.60511.log.gz 2024-09-17T23:41:51.632 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60354.log 2024-09-17T23:41:51.632 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.57171.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.57171.log.gz 2024-09-17T23:41:51.632 INFO:teuthology.orchestra.run.smithi038.stderr: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.53261.log.gz 2024-09-17T23:41:51.635 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.71504.log 2024-09-17T23:41:51.635 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.60354.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36560.log 2024-09-17T23:41:51.635 INFO:teuthology.orchestra.run.smithi038.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.60354.log.gz 2024-09-17T23:41:51.635 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.admin.71504.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.71504.log.gz 2024-09-17T23:41:51.635 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52049.log 2024-09-17T23:41:51.635 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.admin.36560.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36560.log.gz 2024-09-17T23:41:51.636 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39335.log 2024-09-17T23:41:51.638 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.52049.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.52049.log.gz 2024-09-17T23:41:51.638 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54567.log 2024-09-17T23:41:51.638 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.admin.39335.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39308.log 2024-09-17T23:41:51.638 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose/var/log/ceph/cluster1-client.mirror.54567.log: -- /var/log/ceph/cluster2-client.mirror.62618.log 2024-09-17T23:41:51.638 INFO:teuthology.orchestra.run.smithi038.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.54567.log.gz 2024-09-17T23:41:51.642 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.admin.39308.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57583.log 2024-09-17T23:41:51.643 INFO:teuthology.orchestra.run.smithi038.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.42357.log.gz 2024-09-17T23:41:51.643 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.62618.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49451.log 2024-09-17T23:41:51.643 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.57583.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.62618.log.gz 2024-09-17T23:41:51.643 INFO:teuthology.orchestra.run.smithi038.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.57583.log.gz 2024-09-17T23:41:51.643 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66100.log 2024-09-17T23:41:51.643 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.49451.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48807.log 2024-09-17T23:41:51.643 INFO:teuthology.orchestra.run.smithi038.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.49451.log.gz 2024-09-17T23:41:51.643 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.66100.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52917.log 2024-09-17T23:41:51.643 INFO:teuthology.orchestra.run.smithi038.stderr: 86.9%/var/log/ceph/cluster2-client.mirror.48807.log: -- replaced with /var/log/ceph/cluster1-client.mirror.66100.log.gz 2024-09-17T23:41:51.643 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36392.log 2024-09-17T23:41:51.645 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.52917.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.52917.log.gz 2024-09-17T23:41:51.646 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65718.log 2024-09-17T23:41:51.646 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.admin.36392.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36392.log.gz 2024-09-17T23:41:51.646 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60579.log 2024-09-17T23:41:51.646 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.65718.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.65718.log.gz 2024-09-17T23:41:51.648 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62287.log 2024-09-17T23:41:51.648 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.60579.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.60579.log.gz 2024-09-17T23:41:51.648 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51953.log 2024-09-17T23:41:51.651 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.62287.log: 90.2% -- replaced with /var/log/ceph/cluster2-client.mirror.62287.log.gz 2024-09-17T23:41:51.651 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45725.log 2024-09-17T23:41:51.651 INFO:teuthology.orchestra.run.smithi038.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.39308.log.gz 2024-09-17T23:41:51.652 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.51953.log: 87.1% -- replaced with /var/log/ceph/cluster1-client.admin.39335.log.gz 2024-09-17T23:41:51.653 INFO:teuthology.orchestra.run.smithi038.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.51953.log.gz 2024-09-17T23:41:51.653 INFO:teuthology.orchestra.run.smithi038.stderr: 88.4% -- replaced with /var/log/ceph/cluster1-client.mirror.40938.log.gz 2024-09-17T23:41:51.653 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52112.log 2024-09-17T23:41:51.653 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.45725.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.45725.log.gz 2024-09-17T23:41:51.654 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71155.log 2024-09-17T23:41:51.655 INFO:teuthology.orchestra.run.smithi038.stderr: 92.7% -- replaced with /var/log/ceph/cluster2.audit.log.gz 2024-09-17T23:41:51.655 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.52112.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.52112.log.gz 2024-09-17T23:41:51.656 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62701.log 2024-09-17T23:41:51.656 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.71155.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.71155.log.gz 2024-09-17T23:41:51.657 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67411.log 2024-09-17T23:41:51.658 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.62701.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.62701.log.gz 2024-09-17T23:41:51.658 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59541.log 2024-09-17T23:41:51.678 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.67411.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62411.log 2024-09-17T23:41:51.680 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.59541.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.59541.log.gz 2024-09-17T23:41:51.684 INFO:teuthology.orchestra.run.smithi038.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.45262.log.gz 2024-09-17T23:41:51.685 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50762.log 2024-09-17T23:41:51.688 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.62411.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.62411.log.gz 2024-09-17T23:41:51.688 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46117.log 2024-09-17T23:41:51.690 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.50762.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50762.log.gz 2024-09-17T23:41:51.690 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63923.log 2024-09-17T23:41:51.692 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.46117.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46117.log.gz 2024-09-17T23:41:51.692 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59274.log 2024-09-17T23:41:51.692 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.63923.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.63923.log.gz 2024-09-17T23:41:51.693 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57652.log 2024-09-17T23:41:51.695 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.59274.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.59274.log.gz 2024-09-17T23:41:51.696 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58806.log 2024-09-17T23:41:51.699 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.57652.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.57652.log.gz 2024-09-17T23:41:51.699 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70676.log 2024-09-17T23:41:51.700 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.58806.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.58806.log.gz 2024-09-17T23:41:51.700 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55319.log 2024-09-17T23:41:51.701 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.70676.log: 82.8% -- replaced with /var/log/ceph/cluster2-client.mirror.70676.log.gz 2024-09-17T23:41:51.702 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63220.log 2024-09-17T23:41:51.704 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.55319.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.55319.log.gz 2024-09-17T23:41:51.704 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50217.log 2024-09-17T23:41:51.705 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.63220.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.63220.log.gz 2024-09-17T23:41:51.705 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46096.log 2024-09-17T23:41:51.707 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.50217.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.50217.log.gz 2024-09-17T23:41:51.707 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63675.log 2024-09-17T23:41:51.719 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.46096.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57388.log 2024-09-17T23:41:51.720 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.63675.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.63675.log.gz 2024-09-17T23:41:51.725 INFO:teuthology.orchestra.run.smithi038.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48053.log.gz 2024-09-17T23:41:51.726 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64047.log 2024-09-17T23:41:51.727 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.57388.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.57388.log.gz 2024-09-17T23:41:51.727 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41652.log 2024-09-17T23:41:51.728 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.64047.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.64047.log.gz 2024-09-17T23:41:51.728 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60402.log 2024-09-17T23:41:51.729 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.admin.41652.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64749.log 2024-09-17T23:41:51.730 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.60402.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.60402.log.gz 2024-09-17T23:41:51.731 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45075.log 2024-09-17T23:41:51.734 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.64749.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.64749.log.gz 2024-09-17T23:41:51.734 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55970.log 2024-09-17T23:41:51.735 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.45075.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.45075.log.gz 2024-09-17T23:41:51.735 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61932.log 2024-09-17T23:41:51.737 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.55970.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.55970.log.gz 2024-09-17T23:41:51.737 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45865.log 2024-09-17T23:41:51.740 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.61932.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.61932.log.gz 2024-09-17T23:41:51.741 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59880.log 2024-09-17T23:41:51.742 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.45865.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.45865.log.gz 2024-09-17T23:41:51.743 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55715.log 2024-09-17T23:41:51.744 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.59880.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.59880.log.gz 2024-09-17T23:41:51.744 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36518.log 2024-09-17T23:41:51.745 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.55715.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.55715.log.gz 2024-09-17T23:41:51.746 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64687.log 2024-09-17T23:41:51.747 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.admin.36518.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36518.log.gz 2024-09-17T23:41:51.747 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.47425.log 2024-09-17T23:41:51.749 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.64687.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.46176.log.gz 2024-09-17T23:41:51.749 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45446.log 2024-09-17T23:41:51.750 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.2.47425.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.64687.log.gz 2024-09-17T23:41:51.750 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38737.log 2024-09-17T23:41:51.751 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.45446.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.45446.log.gz 2024-09-17T23:41:51.751 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45747.log 2024-09-17T23:41:51.752 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.admin.38737.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58219.log 2024-09-17T23:41:51.754 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.45747.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.2.47425.log.gz 2024-09-17T23:41:51.755 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50603.log 2024-09-17T23:41:51.755 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.58219.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.58219.log.gz 2024-09-17T23:41:51.757 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.38737.log.gz 2024-09-17T23:41:51.757 INFO:teuthology.orchestra.run.smithi038.stderr: /var/log/ceph/cluster2-client.mirror.44225.log 2024-09-17T23:41:51.757 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.50603.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.50603.log.gz 2024-09-17T23:41:51.757 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61637.log 2024-09-17T23:41:51.763 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.44225.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.44225.log.gz 2024-09-17T23:41:51.764 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66955.log 2024-09-17T23:41:51.765 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.61637.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.61637.log.gz 2024-09-17T23:41:51.765 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57041.log 2024-09-17T23:41:51.766 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.66955.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.66955.log.gz 2024-09-17T23:41:51.766 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52367.log 2024-09-17T23:41:51.768 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.57041.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.57041.log.gz 2024-09-17T23:41:51.768 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55556.log 2024-09-17T23:41:51.770 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.52367.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.52367.log.gz 2024-09-17T23:41:51.770 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69270.log 2024-09-17T23:41:51.771 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.55556.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.55556.log.gz 2024-09-17T23:41:51.771 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58674.log 2024-09-17T23:41:51.772 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.69270.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.69270.log.gz 2024-09-17T23:41:51.773 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51591.log 2024-09-17T23:41:51.776 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.58674.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.58674.log.gz 2024-09-17T23:41:51.776 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42912.log 2024-09-17T23:41:51.776 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.51591.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.51591.log.gz 2024-09-17T23:41:51.777 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62825.log 2024-09-17T23:41:51.784 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.42912.log: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.42912.log.gz 2024-09-17T23:41:51.784 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59049.log 2024-09-17T23:41:51.784 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.62825.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.62825.log.gz 2024-09-17T23:41:51.785 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40822.log 2024-09-17T23:41:51.788 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.59049.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.59049.log.gz 2024-09-17T23:41:51.788 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39111.log 2024-09-17T23:41:51.789 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.40822.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59703.log 2024-09-17T23:41:51.789 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.39111.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45203.log 2024-09-17T23:41:51.790 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.59703.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.59703.log.gz 2024-09-17T23:41:51.791 INFO:teuthology.orchestra.run.smithi038.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.40822.log.gz 2024-09-17T23:41:51.791 INFO:teuthology.orchestra.run.smithi038.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.39111.log.gz 2024-09-17T23:41:51.793 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43232.log 2024-09-17T23:41:51.793 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.45203.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.45203.log.gz 2024-09-17T23:41:51.793 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56448.log 2024-09-17T23:41:51.795 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.43232.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.43232.log.gz 2024-09-17T23:41:51.795 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45645.log 2024-09-17T23:41:51.796 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.56448.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.56448.log.gz 2024-09-17T23:41:51.797 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64811.log 2024-09-17T23:41:51.797 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.45645.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.41652.log.gz 2024-09-17T23:41:51.798 INFO:teuthology.orchestra.run.smithi038.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.45645.log.gz 2024-09-17T23:41:51.798 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70851.log 2024-09-17T23:41:51.801 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.64811.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.64811.log.gz 2024-09-17T23:41:51.802 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40474.log 2024-09-17T23:41:51.805 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.70851.log: 91.1% -- replaced with /var/log/ceph/cluster2-client.mirror.70851.log.gz 2024-09-17T23:41:51.805 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41026.log 2024-09-17T23:41:51.806 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.40474.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52524.log 2024-09-17T23:41:51.807 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.41026.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64584.log 2024-09-17T23:41:51.809 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.52524.log: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.52524.log.gz 2024-09-17T23:41:51.810 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63323.log 2024-09-17T23:41:51.811 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.64584.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.64584.log.gz 2024-09-17T23:41:51.811 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58412.log 2024-09-17T23:41:51.814 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.63323.log: 90.2% -- replaced with /var/log/ceph/cluster2-client.mirror.63323.log.gz 2024-09-17T23:41:51.815 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47147.log 2024-09-17T23:41:51.815 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.58412.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.58412.log.gz 2024-09-17T23:41:51.816 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37535.log 2024-09-17T23:41:51.818 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.47147.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47147.log.gz 2024-09-17T23:41:51.818 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51895.log 2024-09-17T23:41:51.819 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.admin.37535.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68209.log 2024-09-17T23:41:51.821 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.51895.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.51895.log.gz 2024-09-17T23:41:51.821 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36434.log 2024-09-17T23:41:51.822 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.68209.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.68209.log.gz 2024-09-17T23:41:51.822 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66640.log 2024-09-17T23:41:51.823 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.admin.36434.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36434.log.gz 2024-09-17T23:41:51.823 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49105.log 2024-09-17T23:41:51.825 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.66640.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66640.log.gz 2024-09-17T23:41:51.825 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- 79.9% /var/log/ceph/cluster2-client.mirror.58499.log 2024-09-17T23:41:51.825 INFO:teuthology.orchestra.run.smithi038.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.37535.log.gz 2024-09-17T23:41:51.825 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.49105.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.49105.log.gz 2024-09-17T23:41:51.826 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51519.log 2024-09-17T23:41:51.828 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.58499.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.58499.log.gz 2024-09-17T23:41:51.828 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66438.log 2024-09-17T23:41:51.830 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.51519.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.51519.log.gz 2024-09-17T23:41:51.830 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68820.log 2024-09-17T23:41:51.831 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.66438.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66438.log.gz 2024-09-17T23:41:51.831 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44872.log 2024-09-17T23:41:51.832 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.68820.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.68820.log.gz 2024-09-17T23:41:51.832 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63303.log 2024-09-17T23:41:51.834 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.44872.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44872.log.gz 2024-09-17T23:41:51.834 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46293.log 2024-09-17T23:41:51.835 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.63303.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.63303.log.gz 2024-09-17T23:41:51.835 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57607.log 2024-09-17T23:41:51.837 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.46293.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.46293.log.gz 2024-09-17T23:41:51.837 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57559.log 2024-09-17T23:41:51.838 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.57607.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.57607.log.gz 2024-09-17T23:41:51.839 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53956.log 2024-09-17T23:41:51.840 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.57559.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.57559.log.gz 2024-09-17T23:41:51.840 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37620.log 2024-09-17T23:41:51.843 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.53956.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.40474.log.gz 2024-09-17T23:41:51.845 INFO:teuthology.orchestra.run.smithi038.stderr: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.53956.log.gz 2024-09-17T23:41:51.846 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49935.log 2024-09-17T23:41:51.846 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.37620.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68453.log 2024-09-17T23:41:51.848 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.49935.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49935.log.gz 2024-09-17T23:41:51.848 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66550.log 2024-09-17T23:41:51.853 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.68453.log: 56.4% -- replaced with /var/log/ceph/cluster2-client.mirror.37620.log.gz 2024-09-17T23:41:51.853 INFO:teuthology.orchestra.run.smithi038.stderr: 82.9% -- replaced with /var/log/ceph/cluster2-client.mirror.68453.log.gz 2024-09-17T23:41:51.853 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36476.log 2024-09-17T23:41:51.854 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.66550.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66550.log.gz 2024-09-17T23:41:51.855 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60627.log 2024-09-17T23:41:51.855 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.admin.36476.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36476.log.gz 2024-09-17T23:41:51.855 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54371.log 2024-09-17T23:41:51.857 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.60627.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.60627.log.gz 2024-09-17T23:41:51.857 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61678.log 2024-09-17T23:41:51.860 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.54371.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.45182.log.gz 2024-09-17T23:41:51.861 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64460.log 2024-09-17T23:41:51.862 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.61678.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.54371.log.gz 2024-09-17T23:41:51.862 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40453.log 2024-09-17T23:41:51.863 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.64460.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.61678.log.gz 2024-09-17T23:41:51.865 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40517.log 2024-09-17T23:41:51.865 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.40453.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.64460.log.gz 2024-09-17T23:41:51.865 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65898.log 2024-09-17T23:41:51.865 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.40517.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52869.log 2024-09-17T23:41:51.865 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47952.log 2024-09-17T23:41:51.866 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.65898.log: /var/log/ceph/cluster1-client.mirror.52869.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.65898.log.gz 2024-09-17T23:41:51.866 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59091.log 2024-09-17T23:41:51.866 INFO:teuthology.orchestra.run.smithi038.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.52869.log.gz 2024-09-17T23:41:51.867 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.59091.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.40453.log.gz 2024-09-17T23:41:51.868 INFO:teuthology.orchestra.run.smithi038.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.59091.log.gz 2024-09-17T23:41:51.870 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66663.log 2024-09-17T23:41:51.870 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51003.log 2024-09-17T23:41:51.871 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.47952.log: /var/log/ceph/cluster1-client.mirror.66663.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47952.log.gz 2024-09-17T23:41:51.872 INFO:teuthology.orchestra.run.smithi038.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66663.log.gz 2024-09-17T23:41:51.872 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47291.log 2024-09-17T23:41:51.872 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51157.log 2024-09-17T23:41:51.872 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.51003.log: /var/log/ceph/cluster2-client.mirror.47291.log: 80.2% -- replaced with /var/log/ceph/cluster2-client.mirror.47291.log.gz 2024-09-17T23:41:51.873 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42128.log 2024-09-17T23:41:51.873 INFO:teuthology.orchestra.run.smithi038.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51003.log.gz 2024-09-17T23:41:51.873 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.51157.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.51157.log.gz 2024-09-17T23:41:51.877 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36012.log 2024-09-17T23:41:51.877 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.admin.42128.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66573.log 2024-09-17T23:41:51.877 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.admin.36012.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36012.log.gz 2024-09-17T23:41:51.877 INFO:teuthology.orchestra.run.smithi038.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.42128.log.gz 2024-09-17T23:41:51.877 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47270.log 2024-09-17T23:41:51.877 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.66573.log: 86.9%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52387.log 2024-09-17T23:41:51.877 INFO:teuthology.orchestra.run.smithi038.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.66573.log.gz 2024-09-17T23:41:51.877 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42775.log 2024-09-17T23:41:51.878 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.47270.log: /var/log/ceph/cluster2-client.mirror.52387.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.52387.log.gz 2024-09-17T23:41:51.880 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59430.log 2024-09-17T23:41:51.880 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.42775.log: 89.1% -- replaced with /var/log/ceph/cluster1-client.mirror.42775.log.gz 2024-09-17T23:41:51.880 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56390.log 2024-09-17T23:41:51.880 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.59430.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.59430.log.gz 2024-09-17T23:41:51.880 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51326.log 2024-09-17T23:41:51.880 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.56390.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.56390.log.gz 2024-09-17T23:41:51.882 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54588.log 2024-09-17T23:41:51.882 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.51326.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.51326.log.gz 2024-09-17T23:41:51.883 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71336.log 2024-09-17T23:41:51.887 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.54588.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.54588.log.gz 2024-09-17T23:41:51.889 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53674.log 2024-09-17T23:41:51.889 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.71336.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.71336.log.gz 2024-09-17T23:41:51.889 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63778.log 2024-09-17T23:41:51.890 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.53674.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.53674.log.gz 2024-09-17T23:41:51.890 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54519.log 2024-09-17T23:41:51.891 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.63778.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.63778.log.gz 2024-09-17T23:41:51.891 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48437.log 2024-09-17T23:41:51.892 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.54519.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.54519.log.gz 2024-09-17T23:41:51.894 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55064.log 2024-09-17T23:41:51.895 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.48437.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48437.log.gz 2024-09-17T23:41:51.896 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54688.log 2024-09-17T23:41:51.897 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.55064.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.55064.log.gz 2024-09-17T23:41:51.897 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69441.log 2024-09-17T23:41:51.899 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.54688.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.41026.log.gz 2024-09-17T23:41:51.899 INFO:teuthology.orchestra.run.smithi038.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.54688.log.gz 2024-09-17T23:41:51.900 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37710.log 2024-09-17T23:41:51.900 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.69441.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.69441.log.gz 2024-09-17T23:41:51.901 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66708.log 2024-09-17T23:41:51.902 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.37710.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.37710.log.gz 2024-09-17T23:41:51.902 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47573.log 2024-09-17T23:41:51.903 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.66708.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66708.log.gz 2024-09-17T23:41:51.903 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42337.log 2024-09-17T23:41:51.905 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.47573.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47573.log.gz 2024-09-17T23:41:51.905 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63447.log 2024-09-17T23:41:51.905 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.42337.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43752.log 2024-09-17T23:41:51.909 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.63447.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.63447.log.gz 2024-09-17T23:41:51.909 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50022.log 2024-09-17T23:41:51.915 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.43752.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.43752.log.gz 2024-09-17T23:41:51.915 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45505.log 2024-09-17T23:41:51.916 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.50022.log: 1.3% -- replaced with /var/log/ceph/cluster2-client.mirror.50022.log.gz 2024-09-17T23:41:51.916 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63985.log 2024-09-17T23:41:51.927 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.45505.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64894.log 2024-09-17T23:41:51.928 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.63985.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.63985.log.gz 2024-09-17T23:41:51.931 INFO:teuthology.orchestra.run.smithi038.stderr: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.42337.log.gz 2024-09-17T23:41:51.944 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.48659.log 2024-09-17T23:41:51.945 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.64894.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.64894.log.gz 2024-09-17T23:41:51.945 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44712.log 2024-09-17T23:41:51.956 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.1.48659.log: 93.2%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69859.log 2024-09-17T23:41:51.957 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.44712.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.44712.log.gz 2024-09-17T23:41:51.963 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62763.log 2024-09-17T23:41:51.968 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.69859.log: -- replaced with /var/log/ceph/cluster1-client.mirror.1.48659.log.gz 2024-09-17T23:41:51.980 INFO:teuthology.orchestra.run.smithi038.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49977.log.gz 2024-09-17T23:41:51.981 INFO:teuthology.orchestra.run.smithi038.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.40517.log.gz 2024-09-17T23:41:51.982 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39479.log 2024-09-17T23:41:51.983 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.62763.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.62763.log.gz 2024-09-17T23:41:51.997 INFO:teuthology.orchestra.run.smithi038.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48807.log.gz 2024-09-17T23:41:51.998 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35417.log 2024-09-17T23:41:51.999 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.39479.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46657.log 2024-09-17T23:41:51.999 INFO:teuthology.orchestra.run.smithi038.stderr: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.39479.log.gz 2024-09-17T23:41:52.000 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.admin.35417.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45463.log 2024-09-17T23:41:52.018 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.46657.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38879.log 2024-09-17T23:41:52.020 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.45463.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45463.log.gz 2024-09-17T23:41:52.021 INFO:teuthology.orchestra.run.smithi038.stderr: 80.3% -- replaced with /var/log/ceph/cluster2-client.admin.35417.log.gz 2024-09-17T23:41:52.037 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58605.log 2024-09-17T23:41:52.043 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.38879.log: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.38879.log.gz 2024-09-17T23:41:52.045 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66730.log 2024-09-17T23:41:52.047 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.58605.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.58605.log.gz 2024-09-17T23:41:52.058 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49149.log 2024-09-17T23:41:52.059 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.66730.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66730.log.gz 2024-09-17T23:41:52.072 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43324.log 2024-09-17T23:41:52.072 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.49149.log: 80.2% -- replaced with /var/log/ceph/cluster2-client.mirror.49149.log.gz 2024-09-17T23:41:52.081 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36809.log 2024-09-17T23:41:52.090 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.43324.log: 89.5%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51678.log 2024-09-17T23:41:52.095 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.36809.log: -- replaced with /var/log/ceph/cluster1-client.mirror.43324.log.gz 2024-09-17T23:41:52.108 INFO:teuthology.orchestra.run.smithi038.stderr: 89.1% -- replaced with /var/log/ceph/cluster1-client.mirror.36809.log.gz 2024-09-17T23:41:52.110 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49748.log 2024-09-17T23:41:52.121 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.51678.log: 89.5%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63427.log 2024-09-17T23:41:52.122 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.49748.log: -- replaced with /var/log/ceph/cluster2-client.mirror.51678.log.gz 2024-09-17T23:41:52.130 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68374.log 2024-09-17T23:41:52.133 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.63427.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.63427.log.gz 2024-09-17T23:41:52.149 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60153.log 2024-09-17T23:41:52.150 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.68374.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.68374.log.gz 2024-09-17T23:41:52.158 INFO:teuthology.orchestra.run.smithi038.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.46096.log.gz 2024-09-17T23:41:52.159 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44245.log 2024-09-17T23:41:52.161 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.60153.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.60153.log.gz 2024-09-17T23:41:52.161 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46575.log 2024-09-17T23:41:52.167 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.44245.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.44245.log.gz 2024-09-17T23:41:52.167 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50844.log 2024-09-17T23:41:52.180 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.46575.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47248.log 2024-09-17T23:41:52.181 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.50844.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.50844.log.gz 2024-09-17T23:41:52.194 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69779.log 2024-09-17T23:41:52.196 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.47248.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.47248.log.gz 2024-09-17T23:41:52.196 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65470.log 2024-09-17T23:41:52.203 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.69779.log: 91.1%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64770.log 2024-09-17T23:41:52.203 INFO:teuthology.orchestra.run.smithi038.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.69779.log.gz 2024-09-17T23:41:52.217 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.65470.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.65470.log.gz 2024-09-17T23:41:52.223 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50001.log 2024-09-17T23:41:52.224 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.64770.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.64770.log.gz 2024-09-17T23:41:52.231 INFO:teuthology.orchestra.run.smithi038.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.45747.log.gz 2024-09-17T23:41:52.233 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41090.log 2024-09-17T23:41:52.233 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.50001.log: 1.3% -- replaced with /var/log/ceph/cluster2-client.mirror.50001.log.gz 2024-09-17T23:41:52.242 INFO:teuthology.orchestra.run.smithi038.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.49748.log.gz 2024-09-17T23:41:52.242 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70319.log 2024-09-17T23:41:52.243 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.41090.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.69962.log 2024-09-17T23:41:52.244 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.70319.log: 85.8% -- replaced with /var/log/ceph/cluster1-client.mirror.70319.log.gz 2024-09-17T23:41:52.244 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35336.log 2024-09-17T23:41:52.248 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.1.69962.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.1.69962.log.gz 2024-09-17T23:41:52.248 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64873.log 2024-09-17T23:41:52.248 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.admin.35336.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.35336.log.gz 2024-09-17T23:41:52.249 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45384.log 2024-09-17T23:41:52.251 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.64873.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.64873.log.gz 2024-09-17T23:41:52.251 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50169.log 2024-09-17T23:41:52.252 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.45384.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45384.log.gz 2024-09-17T23:41:52.254 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42584.log 2024-09-17T23:41:52.254 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.50169.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.50169.log.gz 2024-09-17T23:41:52.254 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43620.log 2024-09-17T23:41:52.256 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.42584.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70931.log 2024-09-17T23:41:52.256 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.43620.log: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.43620.log.gz 2024-09-17T23:41:52.257 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67317.log 2024-09-17T23:41:52.261 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.70931.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.69859.log.gz 2024-09-17T23:41:52.265 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65583.log 2024-09-17T23:41:52.266 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.67317.log: 1.3% -- replaced with /var/log/ceph/cluster1-client.mirror.67317.log.gz 2024-09-17T23:41:52.267 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-osd.0.log 2024-09-17T23:41:52.268 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.65583.log: 86.5% -- replaced with /var/log/ceph/cluster1-client.mirror.65583.log.gz 2024-09-17T23:41:52.268 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61006.log 2024-09-17T23:41:52.269 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-osd.0.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70195.log 2024-09-17T23:41:52.269 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.61006.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.61006.log.gz 2024-09-17T23:41:52.270 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52821.log 2024-09-17T23:41:52.272 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.70195.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.70195.log.gz 2024-09-17T23:41:52.272 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62078.log 2024-09-17T23:41:52.272 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.52821.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.52821.log.gz 2024-09-17T23:41:52.273 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45160.log 2024-09-17T23:41:52.275 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.62078.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.62078.log.gz 2024-09-17T23:41:52.275 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50675.log 2024-09-17T23:41:52.275 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.45160.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.45160.log.gz 2024-09-17T23:41:52.276 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35632.log 2024-09-17T23:41:52.277 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.50675.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.50675.log.gz 2024-09-17T23:41:52.278 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40185.log 2024-09-17T23:41:52.279 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.admin.35632.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.35632.log.gz 2024-09-17T23:41:52.279 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68779.log 2024-09-17T23:41:52.280 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.40185.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48400.log 2024-09-17T23:41:52.281 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.68779.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.68779.log.gz 2024-09-17T23:41:52.281 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53017.log 2024-09-17T23:41:52.282 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.48400.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.48400.log.gz 2024-09-17T23:41:52.283 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37569.log 2024-09-17T23:41:52.283 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.53017.log: 73.4% -- replaced with /var/log/ceph/cluster1-client.mirror.53017.log.gz 2024-09-17T23:41:52.284 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39268.log 2024-09-17T23:41:52.285 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.37569.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36011.log 2024-09-17T23:41:52.286 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.39268.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59904.log 2024-09-17T23:41:52.287 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.admin.36011.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36011.log.gz 2024-09-17T23:41:52.287 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36308.log 2024-09-17T23:41:52.288 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.59904.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.59904.log.gz 2024-09-17T23:41:52.288 INFO:teuthology.orchestra.run.smithi038.stderr: 54.9% -- replaced with /var/log/ceph/cluster2-client.mirror.37569.log.gz 2024-09-17T23:41:52.290 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63241.log 2024-09-17T23:41:52.290 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.admin.36308.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36308.log.gz 2024-09-17T23:41:52.291 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42405.log 2024-09-17T23:41:52.291 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.63241.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.63241.log.gz 2024-09-17T23:41:52.292 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66685.log 2024-09-17T23:41:52.292 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.42405.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57930.log 2024-09-17T23:41:52.293 INFO:teuthology.orchestra.run.smithi038.stderr: 88.8% -- replaced with /var/log/ceph/cluster1-client.mirror.42405.log.gz 2024-09-17T23:41:52.294 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.66685.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.66685.log.gz 2024-09-17T23:41:52.303 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52741.log 2024-09-17T23:41:52.303 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.57930.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.57930.log.gz 2024-09-17T23:41:52.305 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71082.log 2024-09-17T23:41:52.307 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.52741.log: 92.0% -- replaced with /var/log/ceph/cluster1-client.mirror.52741.log.gz 2024-09-17T23:41:52.319 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66528.log 2024-09-17T23:41:52.319 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.71082.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.71082.log.gz 2024-09-17T23:41:52.320 INFO:teuthology.orchestra.run.smithi038.stderr:gzip/var/log/ceph/cluster1-client.mirror.66528.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.66528.log.gz 2024-09-17T23:41:52.323 INFO:teuthology.orchestra.run.smithi038.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46233.log 2024-09-17T23:41:52.323 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67203.log 2024-09-17T23:41:52.323 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.46233.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46233.log.gz 2024-09-17T23:41:52.325 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38916.log 2024-09-17T23:41:52.325 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.67203.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67203.log.gz 2024-09-17T23:41:52.326 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39204.log 2024-09-17T23:41:52.326 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.38916.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69114.log 2024-09-17T23:41:52.327 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.39204.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65920.log 2024-09-17T23:41:52.331 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.69114.log: 74.0% -- replaced with /var/log/ceph/cluster2-client.mirror.38916.log.gz 2024-09-17T23:41:52.332 INFO:teuthology.orchestra.run.smithi038.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.69114.log.gz 2024-09-17T23:41:52.336 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57631.log 2024-09-17T23:41:52.337 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.65920.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.65920.log.gz 2024-09-17T23:41:52.340 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59229.log 2024-09-17T23:41:52.341 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.57631.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.57631.log.gz 2024-09-17T23:41:52.342 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59499.log 2024-09-17T23:41:52.342 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.59229.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.59229.log.gz 2024-09-17T23:41:52.350 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63633.log 2024-09-17T23:41:52.360 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.59499.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.59499.log.gz 2024-09-17T23:41:52.360 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46516.log 2024-09-17T23:41:52.363 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.63633.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.63633.log.gz 2024-09-17T23:41:52.364 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58477.log 2024-09-17T23:41:52.364 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.46516.log: 80.2% -- replaced with /var/log/ceph/cluster2-client.mirror.46516.log.gz 2024-09-17T23:41:52.365 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41678.log 2024-09-17T23:41:52.368 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.58477.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.58477.log.gz 2024-09-17T23:41:52.368 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56544.log 2024-09-17T23:41:52.368 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.41678.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48542.log.gz 2024-09-17T23:41:52.368 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56131.log 2024-09-17T23:41:52.369 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose --/var/log/ceph/cluster1-client.mirror.56544.log: /var/log/ceph/cluster2-client.mirror.62597.log 2024-09-17T23:41:52.370 INFO:teuthology.orchestra.run.smithi038.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.56544.log.gz/var/log/ceph/cluster2-client.mirror.56131.log: 2024-09-17T23:41:52.372 INFO:teuthology.orchestra.run.smithi038.stderr: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.56131.log.gz 2024-09-17T23:41:52.372 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54036.log 2024-09-17T23:41:52.375 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.62597.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.62597.log.gz 2024-09-17T23:41:52.375 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40098.log 2024-09-17T23:41:52.378 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.54036.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.54036.log.gz 2024-09-17T23:41:52.378 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54977.log 2024-09-17T23:41:52.379 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44992.log 2024-09-17T23:41:52.379 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.54977.log: /var/log/ceph/cluster1-client.mirror.40098.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49590.log 2024-09-17T23:41:52.379 INFO:teuthology.orchestra.run.smithi038.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.54977.log.gz 2024-09-17T23:41:52.379 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.44992.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40163.log 2024-09-17T23:41:52.381 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.49590.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.44992.log.gz 2024-09-17T23:41:52.381 INFO:teuthology.orchestra.run.smithi038.stderr: 89.4%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42799.log 2024-09-17T23:41:52.381 INFO:teuthology.orchestra.run.smithi038.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.49590.log.gz 2024-09-17T23:41:52.382 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41854.log 2024-09-17T23:41:52.382 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.40163.log: /var/log/ceph/cluster1-client.mirror.42799.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51736.log 2024-09-17T23:41:52.382 INFO:teuthology.orchestra.run.smithi038.stderr: 89.0%/var/log/ceph/cluster2-client.mirror.41854.log: -- replaced with /var/log/ceph/cluster1-client.mirror.42799.log.gz 2024-09-17T23:41:52.382 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48253.log 2024-09-17T23:41:52.383 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53385.log 2024-09-17T23:41:52.385 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.48253.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48253.log.gz 2024-09-17T23:41:52.385 INFO:teuthology.orchestra.run.smithi038.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.41854.log.gz 2024-09-17T23:41:52.388 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.51736.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.51736.log.gz 2024-09-17T23:41:52.388 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52721.log 2024-09-17T23:41:52.388 INFO:teuthology.orchestra.run.smithi038.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.40163.log.gz 2024-09-17T23:41:52.388 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.53385.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45347.log 2024-09-17T23:41:52.388 INFO:teuthology.orchestra.run.smithi038.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.53385.log.gz 2024-09-17T23:41:52.388 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53737.log 2024-09-17T23:41:52.395 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.45347.log: /var/log/ceph/cluster2-client.mirror.52721.log: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.52721.log.gz 2024-09-17T23:41:52.395 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47654.log 2024-09-17T23:41:52.396 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.53737.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.53737.log.gz 2024-09-17T23:41:52.397 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56689.log 2024-09-17T23:41:52.399 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.47654.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47654.log.gz 2024-09-17T23:41:52.399 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45240.log 2024-09-17T23:41:52.400 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.56689.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.56689.log.gz 2024-09-17T23:41:52.400 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70532.log 2024-09-17T23:41:52.401 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.45240.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.45240.log.gz 2024-09-17T23:41:52.401 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61788.log 2024-09-17T23:41:52.402 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.70532.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.70532.log.gz 2024-09-17T23:41:52.403 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61412.log 2024-09-17T23:41:52.404 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.61788.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.61788.log.gz 2024-09-17T23:41:52.404 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65200.log 2024-09-17T23:41:52.405 INFO:teuthology.orchestra.run.smithi038.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.67411.log.gz 2024-09-17T23:41:52.406 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.61412.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.61412.log.gz 2024-09-17T23:41:52.407 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44053.log 2024-09-17T23:41:52.409 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose --/var/log/ceph/cluster1-client.mirror.65200.log: /var/log/ceph/cluster2-client.mirror.57515.log 2024-09-17T23:41:52.409 INFO:teuthology.orchestra.run.smithi038.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.65200.log.gz 2024-09-17T23:41:52.409 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.44053.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39246.log 2024-09-17T23:41:52.410 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.57515.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.57515.log.gz 2024-09-17T23:41:52.411 INFO:teuthology.orchestra.run.smithi038.stderr: 93.4%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55411.log 2024-09-17T23:41:52.411 INFO:teuthology.orchestra.run.smithi038.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.40098.log.gz 2024-09-17T23:41:52.411 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.39246.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60194.log 2024-09-17T23:41:52.413 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.55411.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.55411.log.gz 2024-09-17T23:41:52.413 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70137.log 2024-09-17T23:41:52.416 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.60194.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.39246.log.gz 2024-09-17T23:41:52.416 INFO:teuthology.orchestra.run.smithi038.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.44053.log.gz 2024-09-17T23:41:52.416 INFO:teuthology.orchestra.run.smithi038.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.60194.log.gz 2024-09-17T23:41:52.416 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45483.log 2024-09-17T23:41:52.422 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.70137.log: 91.1% -- replaced with /var/log/ceph/cluster2-client.mirror.70137.log.gz 2024-09-17T23:41:52.422 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41551.log 2024-09-17T23:41:52.422 INFO:teuthology.orchestra.run.smithi038.stderr:gzip/var/log/ceph/cluster2-client.mirror.45483.log: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45885.log 2024-09-17T23:41:52.422 INFO:teuthology.orchestra.run.smithi038.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45483.log.gz 2024-09-17T23:41:52.422 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.41551.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54805.log 2024-09-17T23:41:52.422 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.45885.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54451.log 2024-09-17T23:41:52.422 INFO:teuthology.orchestra.run.smithi038.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.45885.log.gz 2024-09-17T23:41:52.422 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.54805.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69837.log 2024-09-17T23:41:52.424 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.54451.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.54451.log.gz 2024-09-17T23:41:52.424 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51784.log 2024-09-17T23:41:52.424 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.69837.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.69837.log.gz 2024-09-17T23:41:52.425 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71017.log 2024-09-17T23:41:52.427 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.51784.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.51784.log.gz 2024-09-17T23:41:52.427 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68056.log 2024-09-17T23:41:52.427 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.71017.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.71017.log.gz 2024-09-17T23:41:52.428 INFO:teuthology.orchestra.run.smithi038.stderr: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.54805.log.gz 2024-09-17T23:41:52.429 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61521.log 2024-09-17T23:41:52.429 INFO:teuthology.orchestra.run.smithi038.stderr:gzip/var/log/ceph/cluster2-client.mirror.68056.log: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51808.log 2024-09-17T23:41:52.431 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.61521.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.61521.log.gz 2024-09-17T23:41:52.431 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48563.log 2024-09-17T23:41:52.432 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.51808.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.51808.log.gz 2024-09-17T23:41:52.433 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48483.log 2024-09-17T23:41:52.433 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.48563.log: 80.2% -- replaced with /var/log/ceph/cluster2-client.mirror.48563.log.gz 2024-09-17T23:41:52.434 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68274.log 2024-09-17T23:41:52.434 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.48483.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48483.log.gz 2024-09-17T23:41:52.435 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41488.log 2024-09-17T23:41:52.439 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.68274.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68274.log.gz 2024-09-17T23:41:52.442 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47674.log 2024-09-17T23:41:52.451 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.41488.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60081.log 2024-09-17T23:41:52.452 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.47674.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47674.log.gz 2024-09-17T23:41:52.453 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53775.log 2024-09-17T23:41:52.453 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.60081.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.60081.log.gz 2024-09-17T23:41:52.455 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55821.log 2024-09-17T23:41:52.456 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.53775.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.53775.log.gz 2024-09-17T23:41:52.457 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69629.log 2024-09-17T23:41:52.457 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.55821.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.55821.log.gz 2024-09-17T23:41:52.461 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55022.log 2024-09-17T23:41:52.464 INFO:teuthology.orchestra.run.smithi038.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.46657.log.gz 2024-09-17T23:41:52.465 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.69629.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.69629.log.gz 2024-09-17T23:41:52.466 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63530.log 2024-09-17T23:41:52.466 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.55022.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39063.log 2024-09-17T23:41:52.471 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.63530.log: 89.4% 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.45505.log.gz 2024-09-17T23:41:52.471 INFO:teuthology.orchestra.run.smithi038.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.63530.log.gz 2024-09-17T23:41:52.471 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69717.log 2024-09-17T23:41:52.471 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.39063.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68295.log 2024-09-17T23:41:52.471 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.69717.log: gzip -5 --verbose -- 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.69717.log.gz 2024-09-17T23:41:52.471 INFO:teuthology.orchestra.run.smithi038.stderr: /var/log/ceph/cluster1-client.mirror.44523.log 2024-09-17T23:41:52.471 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.68295.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41419.log 2024-09-17T23:41:52.471 INFO:teuthology.orchestra.run.smithi038.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.68295.log.gz 2024-09-17T23:41:52.471 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.44523.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62473.log 2024-09-17T23:41:52.471 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59069.log 2024-09-17T23:41:52.474 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.62473.log: 90.2%/var/log/ceph/cluster1-client.mirror.41419.log: -- replaced with /var/log/ceph/cluster2-client.mirror.62473.log.gz 2024-09-17T23:41:52.474 INFO:teuthology.orchestra.run.smithi038.stderr: 87.4%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.37403.log 2024-09-17T23:41:52.474 INFO:teuthology.orchestra.run.smithi038.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.39063.log.gz 2024-09-17T23:41:52.475 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.59069.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.44523.log.gz 2024-09-17T23:41:52.476 INFO:teuthology.orchestra.run.smithi038.stderr: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.59069.log.gz 2024-09-17T23:41:52.476 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65493.log 2024-09-17T23:41:52.482 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.1.37403.log: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.41551.log.gz 2024-09-17T23:41:52.482 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70453.log 2024-09-17T23:41:52.482 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59253.log 2024-09-17T23:41:52.482 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.70453.log: /var/log/ceph/cluster1-client.mirror.65493.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42453.log 2024-09-17T23:41:52.482 INFO:teuthology.orchestra.run.smithi038.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.65493.log.gz 2024-09-17T23:41:52.482 INFO:teuthology.orchestra.run.smithi038.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.70453.log.gz 2024-09-17T23:41:52.482 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose --/var/log/ceph/cluster1-client.mirror.59253.log: /var/log/ceph/cluster2-client.admin.71456.log 2024-09-17T23:41:52.482 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.42453.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70238.log 2024-09-17T23:41:52.482 INFO:teuthology.orchestra.run.smithi038.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.59253.log.gz 2024-09-17T23:41:52.482 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.admin.71456.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.71456.log.gz 2024-09-17T23:41:52.482 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51041.log 2024-09-17T23:41:52.482 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.70238.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37294.log 2024-09-17T23:41:52.482 INFO:teuthology.orchestra.run.smithi038.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.70238.log.gz 2024-09-17T23:41:52.482 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.51041.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68230.log 2024-09-17T23:41:52.483 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.37294.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51041.log.gz 2024-09-17T23:41:52.490 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48155.log 2024-09-17T23:41:52.490 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.68230.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55628.log 2024-09-17T23:41:52.490 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.48155.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.48155.log.gz 2024-09-17T23:41:52.491 INFO:teuthology.orchestra.run.smithi038.stderr: 91.3%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42058.log 2024-09-17T23:41:52.491 INFO:teuthology.orchestra.run.smithi038.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.68230.log.gz 2024-09-17T23:41:52.491 INFO:teuthology.orchestra.run.smithi038.stderr: 56.7% -- replaced with /var/log/ceph/cluster1-client.mirror.37294.log.gz 2024-09-17T23:41:52.491 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.55628.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45848.log 2024-09-17T23:41:52.491 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.42058.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.55628.log.gz 2024-09-17T23:41:52.491 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65223.log 2024-09-17T23:41:52.491 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55259.log 2024-09-17T23:41:52.491 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.65223.log: /var/log/ceph/cluster2-client.mirror.45848.log: gzip -5 --verbose -- 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.65223.log.gz 2024-09-17T23:41:52.491 INFO:teuthology.orchestra.run.smithi038.stderr: /var/log/ceph/cluster1-client.mirror.1.46960.log 2024-09-17T23:41:52.491 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.55259.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.45848.log.gz 2024-09-17T23:41:52.491 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.36809.log 2024-09-17T23:41:52.491 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.1.46960.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57672.log 2024-09-17T23:41:52.491 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.0.36809.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60306.log 2024-09-17T23:41:52.492 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.57672.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.55259.log.gz 2024-09-17T23:41:52.492 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49328.log 2024-09-17T23:41:52.492 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.60306.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.60306.log.gz 2024-09-17T23:41:52.493 INFO:teuthology.orchestra.run.smithi038.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.1.46960.log.gz 2024-09-17T23:41:52.493 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50740.log 2024-09-17T23:41:52.493 INFO:teuthology.orchestra.run.smithi038.stderr: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.57672.log.gz 2024-09-17T23:41:52.494 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.49328.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42181.log 2024-09-17T23:41:52.494 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.50740.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54234.log 2024-09-17T23:41:52.496 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.42181.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35865.log 2024-09-17T23:41:52.497 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.54234.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.54234.log.gz 2024-09-17T23:41:52.499 INFO:teuthology.orchestra.run.smithi038.stderr: 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.50740.log.gz 2024-09-17T23:41:52.499 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49511.log 2024-09-17T23:41:52.500 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.admin.35865.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.35865.log.gz 2024-09-17T23:41:52.500 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45012.log 2024-09-17T23:41:52.500 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.49511.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49511.log.gz 2024-09-17T23:41:52.501 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64522.log 2024-09-17T23:41:52.502 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.45012.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.45012.log.gz 2024-09-17T23:41:52.504 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60531.log 2024-09-17T23:41:52.507 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.64522.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.64522.log.gz 2024-09-17T23:41:52.507 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43686.log 2024-09-17T23:41:52.507 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.60531.log: 94.0% -- replaced with /var/log/ceph/cluster1-client.mirror.1.37403.log.gz 2024-09-17T23:41:52.507 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.60531.log.gz 2024-09-17T23:41:52.507 INFO:teuthology.orchestra.run.smithi038.stderr: --verbose -- /var/log/ceph/cluster2-client.admin.36140.log 2024-09-17T23:41:52.507 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61613.log 2024-09-17T23:41:52.507 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.admin.36140.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36140.log.gz 2024-09-17T23:41:52.508 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.43686.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45625.log 2024-09-17T23:41:52.508 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.61613.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.61613.log.gz 2024-09-17T23:41:52.509 INFO:teuthology.orchestra.run.smithi038.stderr: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.42181.log.gz 2024-09-17T23:41:52.510 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64026.log 2024-09-17T23:41:52.512 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.45625.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.45625.log.gz 2024-09-17T23:41:52.512 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46496.log 2024-09-17T23:41:52.514 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.64026.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.43686.log.gz 2024-09-17T23:41:52.515 INFO:teuthology.orchestra.run.smithi038.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.64026.log.gz 2024-09-17T23:41:52.516 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62349.log 2024-09-17T23:41:52.516 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.46496.log: gzip -5 --verbose -- /var/log/ceph/cluster2-mon.a.log 2024-09-17T23:41:52.519 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.62349.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.62349.log.gz 2024-09-17T23:41:52.522 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64543.log 2024-09-17T23:41:52.525 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-mon.a.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68714.log 2024-09-17T23:41:52.531 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.64543.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.64543.log.gz 2024-09-17T23:41:52.531 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58171.log 2024-09-17T23:41:52.531 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.68714.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68714.log.gz 2024-09-17T23:41:52.534 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54847.log 2024-09-17T23:41:52.540 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.58171.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.58171.log.gz 2024-09-17T23:41:52.540 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62099.log 2024-09-17T23:41:52.547 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60873.log 2024-09-17T23:41:52.548 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.54847.log: /var/log/ceph/cluster2-client.mirror.62099.log: 88.1% 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.54847.log.gz 2024-09-17T23:41:52.550 INFO:teuthology.orchestra.run.smithi038.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.62099.log.gz 2024-09-17T23:41:52.552 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69056.log 2024-09-17T23:41:52.555 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.60873.log: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.42058.log.gz 2024-09-17T23:41:52.556 INFO:teuthology.orchestra.run.smithi038.stderr: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.60873.log.gz 2024-09-17T23:41:52.556 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67225.log 2024-09-17T23:41:52.556 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.69056.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.69056.log.gz 2024-09-17T23:41:52.557 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70789.log 2024-09-17T23:41:52.559 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.67225.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67225.log.gz 2024-09-17T23:41:52.559 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44952.log 2024-09-17T23:41:52.560 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.70789.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.70789.log.gz 2024-09-17T23:41:52.560 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.36715.log 2024-09-17T23:41:52.561 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.44952.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.44952.log.gz 2024-09-17T23:41:52.562 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43772.log 2024-09-17T23:41:52.563 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.1.36715.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53867.log 2024-09-17T23:41:52.564 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.43772.log: 73.0% -- replaced with /var/log/ceph/cluster1-client.mirror.1.36715.log.gz 2024-09-17T23:41:52.569 INFO:teuthology.orchestra.run.smithi038.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.43772.log.gz 2024-09-17T23:41:52.573 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.44282.log 2024-09-17T23:41:52.573 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.53867.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58457.log 2024-09-17T23:41:52.575 INFO:teuthology.orchestra.run.smithi038.stderr: 87.3%/var/log/ceph/cluster2-client.mirror.0.44282.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.0.44282.log.gz 2024-09-17T23:41:52.576 INFO:teuthology.orchestra.run.smithi038.stderr: 91.5% -- replaced with /var/log/ceph/cluster2-client.mirror.41678.log.gz 2024-09-17T23:41:52.576 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50262.log 2024-09-17T23:41:52.582 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.58457.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.58457.log.gz 2024-09-17T23:41:52.583 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44503.log 2024-09-17T23:41:52.583 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.50262.log: 88.4% -- replaced with /var/log/ceph/cluster1-client.mirror.41419.log.gz 2024-09-17T23:41:52.585 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.53017.log 2024-09-17T23:41:52.585 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.44503.log: -- replaced with /var/log/ceph/cluster1-client.mirror.53867.log.gz 2024-09-17T23:41:52.592 INFO:teuthology.orchestra.run.smithi038.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44503.log.gz 2024-09-17T23:41:52.592 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose 92.4% -- -- replaced with /var/log/ceph/cluster2-client.mirror.50262.log.gz /var/log/ceph/cluster1-client.mirror.55218.log 2024-09-17T23:41:52.592 INFO:teuthology.orchestra.run.smithi038.stderr: 2024-09-17T23:41:52.597 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.2.53017.log: 93.2% -- replaced with /var/log/ceph/cluster2-client.mirror.2.53017.log.gz 2024-09-17T23:41:52.597 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52435.log 2024-09-17T23:41:52.597 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.55218.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.55218.log.gz 2024-09-17T23:41:52.598 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46553.log 2024-09-17T23:41:52.599 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.52435.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.52435.log.gz 2024-09-17T23:41:52.600 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55456.log 2024-09-17T23:41:52.601 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.46553.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46553.log.gz 2024-09-17T23:41:52.601 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63757.log 2024-09-17T23:41:52.604 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.55456.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.55456.log.gz 2024-09-17T23:41:52.604 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62577.log 2024-09-17T23:41:52.606 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.63757.log: 90.2% -- replaced with /var/log/ceph/cluster2-client.mirror.63757.log.gz 2024-09-17T23:41:52.607 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44732.log 2024-09-17T23:41:52.608 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.62577.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.62577.log.gz 2024-09-17T23:41:52.608 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54825.log 2024-09-17T23:41:52.610 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.44732.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.44732.log.gz 2024-09-17T23:41:52.610 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41595.log 2024-09-17T23:41:52.613 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.54825.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58653.log 2024-09-17T23:41:52.615 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.41595.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.46575.log.gz 2024-09-17T23:41:52.615 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51615.log 2024-09-17T23:41:52.618 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.58653.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.58653.log.gz 2024-09-17T23:41:52.618 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57340.log 2024-09-17T23:41:52.618 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.51615.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53361.log 2024-09-17T23:41:52.618 INFO:teuthology.orchestra.run.smithi038.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.51615.log.gz 2024-09-17T23:41:52.618 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.57340.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59679.log 2024-09-17T23:41:52.620 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.53361.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.53361.log.gz 2024-09-17T23:41:52.620 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60465.log 2024-09-17T23:41:52.620 INFO:teuthology.orchestra.run.smithi038.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.57340.log.gz 2024-09-17T23:41:52.620 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.59679.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.59679.log.gz 2024-09-17T23:41:52.621 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50145.log 2024-09-17T23:41:52.621 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.60465.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.60465.log.gz 2024-09-17T23:41:52.622 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48500.log 2024-09-17T23:41:52.623 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.50145.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50145.log.gz 2024-09-17T23:41:52.623 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41277.log 2024-09-17T23:41:52.624 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70175.log 2024-09-17T23:41:52.625 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.48500.log: /var/log/ceph/cluster2-client.mirror.41277.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52170.log 2024-09-17T23:41:52.625 INFO:teuthology.orchestra.run.smithi038.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48500.log.gz 2024-09-17T23:41:52.626 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41834.log 2024-09-17T23:41:52.626 INFO:teuthology.orchestra.run.smithi038.stderr: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.41277.log.gz 2024-09-17T23:41:52.626 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.52170.log: /var/log/ceph/cluster2-client.mirror.70175.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.52170.log.gz 2024-09-17T23:41:52.628 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.70676.log 2024-09-17T23:41:52.628 INFO:teuthology.orchestra.run.smithi038.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.70175.log.gz 2024-09-17T23:41:52.628 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.41834.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69035.log 2024-09-17T23:41:52.628 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37226.log 2024-09-17T23:41:52.629 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.69035.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.41834.log.gz 2024-09-17T23:41:52.629 INFO:teuthology.orchestra.run.smithi038.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.69035.log.gz 2024-09-17T23:41:52.630 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.1.70676.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54132.log 2024-09-17T23:41:52.630 INFO:teuthology.orchestra.run.smithi038.stderr: 92.1% 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.55022.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.54825.log.gz 2024-09-17T23:41:52.630 INFO:teuthology.orchestra.run.smithi038.stderr: 2024-09-17T23:41:52.632 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.admin.37226.log: 66.7% -- replaced with /var/log/ceph/cluster1-client.admin.37226.log.gz 2024-09-17T23:41:52.632 INFO:teuthology.orchestra.run.smithi038.stderr: 91.5% -- replaced with /var/log/ceph/cluster1-client.mirror.1.70676.log.gz 2024-09-17T23:41:52.632 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44589.log 2024-09-17T23:41:52.632 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.54132.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.41715.log 2024-09-17T23:41:52.634 INFO:teuthology.orchestra.run.smithi038.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.54132.log.gz/var/log/ceph/cluster2-client.mirror.44589.log: 2024-09-17T23:41:52.634 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51350.log 2024-09-17T23:41:52.634 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.1.41715.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67870.log 2024-09-17T23:41:52.635 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.51350.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.51350.log.gz 2024-09-17T23:41:52.636 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51461.log 2024-09-17T23:41:52.638 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.67870.log: 91.2% -- replaced with /var/log/ceph/cluster2-client.mirror.67870.log.gz 2024-09-17T23:41:52.638 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41395.log 2024-09-17T23:41:52.639 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.51461.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44589.log.gz 2024-09-17T23:41:52.639 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.61095.log 2024-09-17T23:41:52.639 INFO:teuthology.orchestra.run.smithi038.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.51461.log.gz 2024-09-17T23:41:52.641 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51202.log 2024-09-17T23:41:52.641 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.41395.log: /var/log/ceph/cluster2-client.mirror.2.61095.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59181.log 2024-09-17T23:41:52.642 INFO:teuthology.orchestra.run.smithi038.stderr: 88.5% -- replaced with /var/log/ceph/cluster1-client.mirror.41395.log.gz 2024-09-17T23:41:52.643 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.51202.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51202.log.gz 2024-09-17T23:41:52.644 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64501.log 2024-09-17T23:41:52.644 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.59181.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.59181.log.gz 2024-09-17T23:41:52.645 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67997.log 2024-09-17T23:41:52.647 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.64501.log: 89.3% -- replaced with /var/log/ceph/cluster1-client.mirror.42453.log.gz 2024-09-17T23:41:52.648 INFO:teuthology.orchestra.run.smithi038.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.42584.log.gz 2024-09-17T23:41:52.648 INFO:teuthology.orchestra.run.smithi038.stderr: 90.2% -- replaced with /var/log/ceph/cluster2-client.mirror.64501.log.gz 2024-09-17T23:41:52.649 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56303.log 2024-09-17T23:41:52.649 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.67997.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.67997.log.gz 2024-09-17T23:41:52.650 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46213.log 2024-09-17T23:41:52.650 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.56303.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.56303.log.gz 2024-09-17T23:41:52.652 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48748.log 2024-09-17T23:41:52.652 INFO:teuthology.orchestra.run.smithi038.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.2.61095.log.gz 2024-09-17T23:41:52.652 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.46213.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61891.log 2024-09-17T23:41:52.652 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.48748.log: 80.2% -- replaced with /var/log/ceph/cluster2-client.mirror.48748.log.gz 2024-09-17T23:41:52.653 INFO:teuthology.orchestra.run.smithi038.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.46213.log.gz 2024-09-17T23:41:52.655 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69293.log 2024-09-17T23:41:52.656 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose/var/log/ceph/cluster2-client.mirror.61891.log: -- /var/log/ceph/cluster2-client.mirror.50579.log 2024-09-17T23:41:52.657 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.69293.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.61891.log.gz 2024-09-17T23:41:52.657 INFO:teuthology.orchestra.run.smithi038.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.69293.log.gz 2024-09-17T23:41:52.657 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36850.log 2024-09-17T23:41:52.658 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.50579.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50579.log.gz 2024-09-17T23:41:52.659 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50193.log 2024-09-17T23:41:52.659 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.36850.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48765.log 2024-09-17T23:41:52.660 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.50193.log: 89.1% -- replaced with /var/log/ceph/cluster1-client.mirror.36850.log.gz 2024-09-17T23:41:52.662 INFO:teuthology.orchestra.run.smithi038.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.50193.log.gz 2024-09-17T23:41:52.662 INFO:teuthology.orchestra.run.smithi038.stderr: 92.1% -- replaced with /var/log/ceph/cluster2-client.mirror.0.36809.log.gz 2024-09-17T23:41:52.662 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49918.log 2024-09-17T23:41:52.662 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.48765.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63943.log 2024-09-17T23:41:52.662 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.49918.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.49918.log.gz 2024-09-17T23:41:52.662 INFO:teuthology.orchestra.run.smithi038.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48765.log.gz 2024-09-17T23:41:52.663 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63468.log 2024-09-17T23:41:52.663 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57099.log 2024-09-17T23:41:52.664 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.63943.log: /var/log/ceph/cluster2-client.mirror.63468.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.63468.log.gz 2024-09-17T23:41:52.665 INFO:teuthology.orchestra.run.smithi038.stderr: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.63943.log.gz 2024-09-17T23:41:52.665 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52001.log 2024-09-17T23:41:52.666 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.57099.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.57099.log.gz 2024-09-17T23:41:52.666 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46808.log 2024-09-17T23:41:52.667 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.52001.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.52001.log.gz 2024-09-17T23:41:52.668 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51567.log 2024-09-17T23:41:52.672 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.46808.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45827.log 2024-09-17T23:41:52.672 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.51567.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.51567.log.gz 2024-09-17T23:41:52.673 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48417.log 2024-09-17T23:41:52.697 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.45827.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.45347.log.gz 2024-09-17T23:41:52.699 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52958.log 2024-09-17T23:41:52.703 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.48417.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48417.log.gz 2024-09-17T23:41:52.703 INFO:teuthology.orchestra.run.smithi038.stderr: 94.1% -- replaced with /var/log/ceph/cluster2-client.mirror.1.41715.log.gz 2024-09-17T23:41:52.712 INFO:teuthology.orchestra.run.smithi038.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.39204.log.gz 2024-09-17T23:41:52.718 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43924.log 2024-09-17T23:41:52.721 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.52958.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.52958.log.gz 2024-09-17T23:41:52.728 INFO:teuthology.orchestra.run.smithi038.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.70931.log.gz 2024-09-17T23:41:52.729 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70595.log 2024-09-17T23:41:52.734 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.43924.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.43924.log.gz 2024-09-17T23:41:52.736 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-osd.1.log 2024-09-17T23:41:52.736 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.70595.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.70595.log.gz 2024-09-17T23:41:52.737 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55932.log 2024-09-17T23:41:52.737 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-osd.1.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66325.log 2024-09-17T23:41:52.741 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.55932.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.55932.log.gz 2024-09-17T23:41:52.741 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69335.log 2024-09-17T23:41:52.741 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.66325.log: gzip -5 --verbose -- 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66325.log.gz 2024-09-17T23:41:52.741 INFO:teuthology.orchestra.run.smithi038.stderr: /var/log/ceph/cluster2-client.mirror.41327.log 2024-09-17T23:41:52.741 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47818.log 2024-09-17T23:41:52.741 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.41327.log: /var/log/ceph/cluster1-client.mirror.69335.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50342.log 2024-09-17T23:41:52.741 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.47818.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.69335.log.gz 2024-09-17T23:41:52.742 INFO:teuthology.orchestra.run.smithi038.stderr: 92.1% -- replaced with /var/log/ceph/cluster2-client.mirror.47818.log.gz 2024-09-17T23:41:52.743 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.50342.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.41327.log.gz 2024-09-17T23:41:52.744 INFO:teuthology.orchestra.run.smithi038.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50342.log.gz 2024-09-17T23:41:52.744 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47736.log 2024-09-17T23:41:52.744 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35633.log 2024-09-17T23:41:52.746 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.47736.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47736.log.gz 2024-09-17T23:41:52.747 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53240.log 2024-09-17T23:41:52.748 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60129.log 2024-09-17T23:41:52.748 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.admin.35633.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.35633.log.gz 2024-09-17T23:41:52.748 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.53240.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65065.log 2024-09-17T23:41:52.750 INFO:teuthology.orchestra.run.smithi038.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.53240.log.gz/var/log/ceph/cluster1-client.mirror.60129.log: 2024-09-17T23:41:52.750 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62721.log 2024-09-17T23:41:52.750 INFO:teuthology.orchestra.run.smithi038.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.60129.log.gz/var/log/ceph/cluster1-client.mirror.65065.log: 2024-09-17T23:41:52.751 INFO:teuthology.orchestra.run.smithi038.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.65065.log.gz 2024-09-17T23:41:52.751 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37693.log 2024-09-17T23:41:52.751 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.62721.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42935.log 2024-09-17T23:41:52.752 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.37693.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47915.log 2024-09-17T23:41:52.753 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.42935.log: 90.2% -- replaced with /var/log/ceph/cluster2-client.mirror.62721.log.gz 2024-09-17T23:41:52.754 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55890.log 2024-09-17T23:41:52.755 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.47915.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47915.log.gz 2024-09-17T23:41:52.755 INFO:teuthology.orchestra.run.smithi038.stderr: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.37693.log.gz 2024-09-17T23:41:52.755 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46154.log 2024-09-17T23:41:52.759 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.55890.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.55890.log.gz 2024-09-17T23:41:52.759 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67158.log 2024-09-17T23:41:52.759 INFO:teuthology.orchestra.run.smithi038.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.42935.log.gz 2024-09-17T23:41:52.759 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.46154.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37654.log 2024-09-17T23:41:52.759 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.67158.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46154.log.gz 2024-09-17T23:41:52.759 INFO:teuthology.orchestra.run.smithi038.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67158.log.gz 2024-09-17T23:41:52.759 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54278.log 2024-09-17T23:41:52.761 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62804.log 2024-09-17T23:41:52.761 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.54278.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.54278.log.gz 2024-09-17T23:41:52.761 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52604.log 2024-09-17T23:41:52.762 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.62804.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.62804.log.gz 2024-09-17T23:41:52.763 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47108.log 2024-09-17T23:41:52.764 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.52604.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.52604.log.gz 2024-09-17T23:41:52.764 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.37654.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53602.log 2024-09-17T23:41:52.772 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.47108.log: 53.8% -- replaced with /var/log/ceph/cluster2-client.mirror.37654.log.gz 2024-09-17T23:41:52.772 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44752.log 2024-09-17T23:41:52.786 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.53602.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37603.log 2024-09-17T23:41:52.787 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.44752.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.44752.log.gz 2024-09-17T23:41:52.789 INFO:teuthology.orchestra.run.smithi038.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49328.log.gz 2024-09-17T23:41:52.789 INFO:teuthology.orchestra.run.smithi038.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.53602.log.gz 2024-09-17T23:41:52.791 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47049.log 2024-09-17T23:41:52.792 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.37603.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64295.log 2024-09-17T23:41:52.792 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.47049.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47049.log.gz 2024-09-17T23:41:52.792 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60756.log 2024-09-17T23:41:52.793 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.64295.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.64295.log.gz 2024-09-17T23:41:52.794 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56978.log 2024-09-17T23:41:52.794 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.60756.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.60756.log.gz 2024-09-17T23:41:52.795 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49068.log 2024-09-17T23:41:52.795 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.56978.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.56978.log.gz 2024-09-17T23:41:52.797 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42892.log 2024-09-17T23:41:52.797 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.49068.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.49068.log.gz 2024-09-17T23:41:52.797 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53520.log 2024-09-17T23:41:52.797 INFO:teuthology.orchestra.run.smithi038.stderr: 2.3% -- replaced with /var/log/ceph/cluster2-client.mirror.37603.log.gz 2024-09-17T23:41:52.802 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.42892.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40983.log 2024-09-17T23:41:52.803 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.53520.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.42892.log.gz 2024-09-17T23:41:52.804 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48134.log 2024-09-17T23:41:52.805 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.40983.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57216.log 2024-09-17T23:41:52.811 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.48134.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.53520.log.gz 2024-09-17T23:41:52.811 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69458.log 2024-09-17T23:41:52.816 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.57216.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.57216.log.gz 2024-09-17T23:41:52.829 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65673.log 2024-09-17T23:41:52.831 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.69458.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.69458.log.gz 2024-09-17T23:41:52.831 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69400.log 2024-09-17T23:41:52.832 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.65673.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.65673.log.gz 2024-09-17T23:41:52.832 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.70319.log 2024-09-17T23:41:52.833 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.69400.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.69400.log.gz 2024-09-17T23:41:52.834 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45033.log 2024-09-17T23:41:52.840 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.0.70319.log: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.0.70319.log.gz 2024-09-17T23:41:52.841 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66078.log 2024-09-17T23:41:52.842 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.45033.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.45033.log.gz 2024-09-17T23:41:52.843 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.47818.log 2024-09-17T23:41:52.844 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.66078.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66078.log.gz 2024-09-17T23:41:52.844 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45586.log 2024-09-17T23:41:52.853 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.3.47818.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48377.log 2024-09-17T23:41:52.860 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.45586.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53144.log 2024-09-17T23:41:52.864 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.48377.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.47270.log.gz 2024-09-17T23:41:52.865 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45527.log 2024-09-17T23:41:52.865 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.53144.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.53144.log.gz 2024-09-17T23:41:52.866 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67369.log 2024-09-17T23:41:52.866 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.45527.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.45527.log.gz 2024-09-17T23:41:52.866 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55146.log 2024-09-17T23:41:52.868 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.67369.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.67369.log.gz 2024-09-17T23:41:52.869 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59382.log 2024-09-17T23:41:52.870 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.55146.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.55146.log.gz 2024-09-17T23:41:52.870 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58537.log 2024-09-17T23:41:52.870 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.59382.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.59382.log.gz 2024-09-17T23:41:52.871 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63695.log 2024-09-17T23:41:52.872 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.58537.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.58537.log.gz 2024-09-17T23:41:52.873 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40914.log 2024-09-17T23:41:52.876 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.63695.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.63695.log.gz 2024-09-17T23:41:52.876 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57858.log 2024-09-17T23:41:52.877 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.40914.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35552.log 2024-09-17T23:41:52.878 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.57858.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.57858.log.gz 2024-09-17T23:41:52.878 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47211.log 2024-09-17T23:41:52.879 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.admin.35552.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.35552.log.gz 2024-09-17T23:41:52.879 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64067.log 2024-09-17T23:41:52.880 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.47211.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47211.log.gz 2024-09-17T23:41:52.882 INFO:teuthology.orchestra.run.smithi038.stderr: 93.8% -- replaced with /var/log/ceph/cluster1-client.mirror.3.47818.log.gz 2024-09-17T23:41:52.882 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41995.log 2024-09-17T23:41:52.891 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54254.log 2024-09-17T23:41:52.903 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.64067.log: /var/log/ceph/cluster2-client.mirror.41995.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50981.log 2024-09-17T23:41:52.914 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.54254.log: 87.7% 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.64067.log.gz 2024-09-17T23:41:52.915 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50627.log 2024-09-17T23:41:52.916 INFO:teuthology.orchestra.run.smithi038.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.54254.log.gz 2024-09-17T23:41:52.922 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.50981.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.50981.log.gz 2024-09-17T23:41:52.923 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56496.log 2024-09-17T23:41:52.931 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.50627.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.50627.log.gz 2024-09-17T23:41:52.932 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66213.log 2024-09-17T23:41:52.932 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.56496.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.56496.log.gz 2024-09-17T23:41:52.933 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60668.log 2024-09-17T23:41:52.936 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.66213.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.66213.log.gz 2024-09-17T23:41:52.937 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40381.log 2024-09-17T23:41:52.946 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.60668.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51439.log 2024-09-17T23:41:52.950 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.40381.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.60668.log.gz 2024-09-17T23:41:52.958 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50868.log 2024-09-17T23:41:52.959 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.51439.log: 88.3% -- replaced with /var/log/ceph/cluster1-client.mirror.40381.log.gz 2024-09-17T23:41:52.960 INFO:teuthology.orchestra.run.smithi038.stderr: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.51439.log.gz 2024-09-17T23:41:52.968 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50916.log 2024-09-17T23:41:52.981 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.50868.log: 87.4%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41133.log 2024-09-17T23:41:52.981 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.50916.log: -- replaced with /var/log/ceph/cluster1-client.mirror.50868.log.gz 2024-09-17T23:41:52.982 INFO:teuthology.orchestra.run.smithi038.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.50916.log.gz 2024-09-17T23:41:52.990 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40866.log 2024-09-17T23:41:52.994 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.41133.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.68056.log.gz 2024-09-17T23:41:52.997 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63034.log 2024-09-17T23:41:52.998 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.40866.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47328.log 2024-09-17T23:41:53.001 INFO:teuthology.orchestra.run.smithi038.stderr: 73.6% -- replaced with /var/log/ceph/cluster2-client.mirror.41133.log.gz 2024-09-17T23:41:53.003 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.63034.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.63034.log.gz 2024-09-17T23:41:53.003 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69478.log 2024-09-17T23:41:53.011 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.47328.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47328.log.gz 2024-09-17T23:41:53.011 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62887.log 2024-09-17T23:41:53.019 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.69478.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.69478.log.gz 2024-09-17T23:41:53.020 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61054.log 2024-09-17T23:41:53.020 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.62887.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.62887.log.gz 2024-09-17T23:41:53.024 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64191.log 2024-09-17T23:41:53.028 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.61054.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.61054.log.gz 2024-09-17T23:41:53.029 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65808.log 2024-09-17T23:41:53.030 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.64191.log: 93.3% -- replaced with /var/log/ceph/cluster2-client.mirror.40983.log.gz 2024-09-17T23:41:53.032 INFO:teuthology.orchestra.run.smithi038.stderr: 90.2% -- replaced with /var/log/ceph/cluster2-client.mirror.64191.log.gz 2024-09-17T23:41:53.032 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69156.log 2024-09-17T23:41:53.032 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.65808.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.65808.log.gz 2024-09-17T23:41:53.033 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43393.log 2024-09-17T23:41:53.043 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.69156.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37311.log 2024-09-17T23:41:53.053 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.43393.log: 93.4%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46921.log 2024-09-17T23:41:53.055 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.37311.log: -- replaced with /var/log/ceph/cluster2-client.mirror.43393.log.gz 2024-09-17T23:41:53.058 INFO:teuthology.orchestra.run.smithi038.stderr: 57.1% -- replaced with /var/log/ceph/cluster1-client.mirror.37311.log.gz 2024-09-17T23:41:53.075 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42518.log 2024-09-17T23:41:53.084 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.46921.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41443.log 2024-09-17T23:41:53.100 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.42518.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70052.log 2024-09-17T23:41:53.118 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.41443.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49283.log 2024-09-17T23:41:53.118 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.70052.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.70052.log.gz 2024-09-17T23:41:53.119 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54712.log 2024-09-17T23:41:53.120 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.49283.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.49283.log.gz 2024-09-17T23:41:53.133 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71411.log 2024-09-17T23:41:53.133 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.54712.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.54712.log.gz 2024-09-17T23:41:53.147 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59969.log 2024-09-17T23:41:53.147 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.71411.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.71411.log.gz 2024-09-17T23:41:53.171 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63881.log 2024-09-17T23:41:53.173 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.59969.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.59969.log.gz 2024-09-17T23:41:53.183 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43413.log 2024-09-17T23:41:53.199 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.63881.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.63881.log.gz 2024-09-17T23:41:53.204 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47497.log 2024-09-17T23:41:53.210 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.43413.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.43413.log.gz 2024-09-17T23:41:53.213 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65155.log 2024-09-17T23:41:53.213 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.47497.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47497.log.gz 2024-09-17T23:41:53.238 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56151.log 2024-09-17T23:41:53.238 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.65155.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.65155.log.gz 2024-09-17T23:41:53.247 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56737.log 2024-09-17T23:41:53.256 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.56151.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67389.log 2024-09-17T23:41:53.257 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.56737.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.56737.log.gz 2024-09-17T23:41:53.261 INFO:teuthology.orchestra.run.smithi038.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.56151.log.gz 2024-09-17T23:41:53.269 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55170.log 2024-09-17T23:41:53.282 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.67389.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.67389.log.gz 2024-09-17T23:41:53.286 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65358.log 2024-09-17T23:41:53.287 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.55170.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.55170.log.gz 2024-09-17T23:41:53.288 INFO:teuthology.orchestra.run.smithi038.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48377.log.gz 2024-09-17T23:41:53.296 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52938.log 2024-09-17T23:41:53.305 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.65358.log: 86.9%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47593.log 2024-09-17T23:41:53.308 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.52938.log: -- replaced with /var/log/ceph/cluster1-client.mirror.65358.log.gz 2024-09-17T23:41:53.312 INFO:teuthology.orchestra.run.smithi038.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.52938.log.gz 2024-09-17T23:41:53.314 INFO:teuthology.orchestra.run.smithi038.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.46496.log.gz 2024-09-17T23:41:53.317 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65023.log 2024-09-17T23:41:53.318 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.47593.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47593.log.gz 2024-09-17T23:41:53.318 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37586.log 2024-09-17T23:41:53.319 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.65023.log: 2.6% -- replaced with /var/log/ceph/cluster1-client.mirror.65023.log.gz 2024-09-17T23:41:53.331 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64956.log 2024-09-17T23:41:53.332 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.37586.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48988.log 2024-09-17T23:41:53.333 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.64956.log: 2.3% -- replaced with /var/log/ceph/cluster2-client.mirror.37586.log.gz 2024-09-17T23:41:53.333 INFO:teuthology.orchestra.run.smithi038.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.64956.log.gz 2024-09-17T23:41:53.341 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52218.log 2024-09-17T23:41:53.344 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.48988.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42977.log 2024-09-17T23:41:53.345 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.52218.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.52218.log.gz 2024-09-17T23:41:53.348 INFO:teuthology.orchestra.run.smithi038.stderr: 80.1% -- replaced with /var/log/ceph/cluster2-client.mirror.48988.log.gz 2024-09-17T23:41:53.348 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56802.log 2024-09-17T23:41:53.349 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.42977.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.42977.log.gz 2024-09-17T23:41:53.350 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61388.log 2024-09-17T23:41:53.353 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.56802.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.56802.log.gz 2024-09-17T23:41:53.353 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61850.log 2024-09-17T23:41:53.353 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.61388.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.61388.log.gz 2024-09-17T23:41:53.353 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59316.log 2024-09-17T23:41:53.360 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.61850.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.61850.log.gz 2024-09-17T23:41:53.360 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37730.log 2024-09-17T23:41:53.362 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.59316.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.59316.log.gz 2024-09-17T23:41:53.363 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59928.log 2024-09-17T23:41:53.366 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.37730.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52504.log 2024-09-17T23:41:53.366 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.59928.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.59928.log.gz 2024-09-17T23:41:53.372 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67352.log 2024-09-17T23:41:53.372 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.52504.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71102.log 2024-09-17T23:41:53.372 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.67352.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.67352.log.gz 2024-09-17T23:41:53.373 INFO:teuthology.orchestra.run.smithi038.stderr: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.52504.log.gz 2024-09-17T23:41:53.376 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40494.log 2024-09-17T23:41:53.379 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.71102.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.37730.log.gz 2024-09-17T23:41:53.382 INFO:teuthology.orchestra.run.smithi038.stderr: 91.0% -- replaced with /var/log/ceph/cluster2-client.mirror.71102.log.gz 2024-09-17T23:41:53.383 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45907.log 2024-09-17T23:41:53.386 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.40494.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53650.log 2024-09-17T23:41:53.404 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.45907.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43967.log 2024-09-17T23:41:53.404 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.53650.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.53650.log.gz 2024-09-17T23:41:53.423 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62783.log 2024-09-17T23:41:53.428 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.43967.log: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.40494.log.gz 2024-09-17T23:41:53.441 INFO:teuthology.orchestra.run.smithi038.stderr: 93.4%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47556.log 2024-09-17T23:41:53.441 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.62783.log: -- replaced with /var/log/ceph/cluster2-client.mirror.43967.log.gz 2024-09-17T23:41:53.443 INFO:teuthology.orchestra.run.smithi038.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48134.log.gz 2024-09-17T23:41:53.444 INFO:teuthology.orchestra.run.smithi038.stderr: 90.2% -- replaced with /var/log/ceph/cluster2-client.mirror.62783.log.gz 2024-09-17T23:41:53.444 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54084.log 2024-09-17T23:41:53.445 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.47556.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47556.log.gz 2024-09-17T23:41:53.447 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62156.log 2024-09-17T23:41:53.447 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.54084.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.54084.log.gz 2024-09-17T23:41:53.447 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52845.log 2024-09-17T23:41:53.447 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.62156.log: 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.62156.log.gz 2024-09-17T23:41:53.448 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48907.log 2024-09-17T23:41:53.449 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.52845.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.52845.log.gz 2024-09-17T23:41:53.450 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59028.log 2024-09-17T23:41:53.450 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.48907.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.48907.log.gz 2024-09-17T23:41:53.450 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55753.log 2024-09-17T23:41:53.451 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.59028.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.59028.log.gz 2024-09-17T23:41:53.451 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69924.log 2024-09-17T23:41:53.452 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.55753.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.55753.log.gz 2024-09-17T23:41:53.452 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58557.log 2024-09-17T23:41:53.454 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.69924.log: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.69924.log.gz 2024-09-17T23:41:53.454 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54326.log 2024-09-17T23:41:53.455 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.58557.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.58557.log.gz 2024-09-17T23:41:53.456 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57364.log 2024-09-17T23:41:53.456 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.54326.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.54326.log.gz 2024-09-17T23:41:53.457 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36894.log 2024-09-17T23:41:53.458 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.57364.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.57364.log.gz 2024-09-17T23:41:53.458 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39159.log 2024-09-17T23:41:53.459 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.36894.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61870.log 2024-09-17T23:41:53.460 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.39159.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44010.log 2024-09-17T23:41:53.461 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.61870.log: 80.9% -- replaced with /var/log/ceph/cluster1-client.mirror.36894.log.gz 2024-09-17T23:41:53.462 INFO:teuthology.orchestra.run.smithi038.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.39159.log.gz 2024-09-17T23:41:53.464 INFO:teuthology.orchestra.run.smithi038.stderr: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.61870.log.gz 2024-09-17T23:41:53.465 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57147.log 2024-09-17T23:41:53.470 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.44010.log: 93.3% -- replaced with /var/log/ceph/cluster2-client.mirror.44010.log.gz 2024-09-17T23:41:53.471 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49414.log 2024-09-17T23:41:53.472 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.57147.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.57147.log.gz 2024-09-17T23:41:53.473 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70952.log 2024-09-17T23:41:53.473 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.49414.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.49414.log.gz 2024-09-17T23:41:53.474 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51244.log 2024-09-17T23:41:53.474 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.70952.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.70952.log.gz 2024-09-17T23:41:53.475 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.32498.log 2024-09-17T23:41:53.476 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.51244.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.51244.log.gz 2024-09-17T23:41:53.477 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63592.log 2024-09-17T23:41:53.477 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.admin.32498.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.32498.log.gz 2024-09-17T23:41:53.478 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41573.log 2024-09-17T23:41:53.479 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.63592.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.63592.log.gz 2024-09-17T23:41:53.480 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37637.log 2024-09-17T23:41:53.480 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.41573.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57694.log 2024-09-17T23:41:53.481 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.37637.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49166.log 2024-09-17T23:41:53.482 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.57694.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.57694.log.gz 2024-09-17T23:41:53.485 INFO:teuthology.orchestra.run.smithi038.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.46808.log.gz 2024-09-17T23:41:53.485 INFO:teuthology.orchestra.run.smithi038.stderr: 57.7% -- replaced with /var/log/ceph/cluster2-client.mirror.37637.log.gz 2024-09-17T23:41:53.485 INFO:teuthology.orchestra.run.smithi038.stderr: 88.3% -- replaced with /var/log/ceph/cluster1-client.mirror.40914.log.gz 2024-09-17T23:41:53.485 INFO:teuthology.orchestra.run.smithi038.stderr: 89.6% 88.5% -- replaced with /var/log/ceph/cluster1-client.mirror.41443.log.gz 2024-09-17T23:41:53.487 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63489.log 2024-09-17T23:41:53.487 INFO:teuthology.orchestra.run.smithi038.stderr: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.40866.log.gz 2024-09-17T23:41:53.487 INFO:teuthology.orchestra.run.smithi038.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.41573.log.gz 2024-09-17T23:41:53.487 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.49166.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62535.log 2024-09-17T23:41:53.488 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.63489.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.63489.log.gz 2024-09-17T23:41:53.488 INFO:teuthology.orchestra.run.smithi038.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49166.log.gz 2024-09-17T23:41:53.489 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58364.log 2024-09-17T23:41:53.489 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.62535.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46731.log 2024-09-17T23:41:53.489 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.58364.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.58364.log.gz 2024-09-17T23:41:53.490 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46196.log 2024-09-17T23:41:53.490 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.46731.log: 77.5% -- replaced with /var/log/ceph/cluster1-client.mirror.46731.log.gz 2024-09-17T23:41:53.490 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56255.log 2024-09-17T23:41:53.491 INFO:teuthology.orchestra.run.smithi038.stderr: 90.3%/var/log/ceph/cluster2-client.mirror.46196.log: -- replaced with /var/log/ceph/cluster2-client.mirror.62535.log.gz 2024-09-17T23:41:53.491 INFO:teuthology.orchestra.run.smithi038.stderr: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.46196.log.gz 2024-09-17T23:41:53.492 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.62944.log 2024-09-17T23:41:53.492 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50651.log 2024-09-17T23:41:53.492 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.56255.log: /var/log/ceph/cluster2-client.mirror.0.62944.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.56255.log.gz 2024-09-17T23:41:53.493 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60443.log 2024-09-17T23:41:53.494 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.50651.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.50651.log.gz 2024-09-17T23:41:53.497 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.60443.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.60443.log.gz 2024-09-17T23:41:53.497 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54108.log 2024-09-17T23:41:53.497 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.71865.log 2024-09-17T23:41:53.498 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.54108.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.54108.log.gz 2024-09-17T23:41:53.503 INFO:teuthology.orchestra.run.smithi038.stderr: 93.2% -- replaced with /var/log/ceph/cluster2-client.mirror.0.62944.log.gz 2024-09-17T23:41:53.504 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66145.log 2024-09-17T23:41:53.507 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.admin.71865.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40210.log 2024-09-17T23:41:53.507 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.71865.log.gz 2024-09-17T23:41:53.508 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.66145.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50559.log 2024-09-17T23:41:53.508 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.admin.40210.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.66145.log.gz 2024-09-17T23:41:53.508 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46356.log 2024-09-17T23:41:53.508 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.50559.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64233.log 2024-09-17T23:41:53.508 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.46356.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50559.log.gz 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46356.log.gz 2024-09-17T23:41:53.508 INFO:teuthology.orchestra.run.smithi038.stderr: 2024-09-17T23:41:53.508 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65785.log 2024-09-17T23:41:53.511 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47696.log 2024-09-17T23:41:53.511 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.65785.log: /var/log/ceph/cluster2-client.mirror.64233.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.65785.log.gz 2024-09-17T23:41:53.511 INFO:teuthology.orchestra.run.smithi038.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.64233.log.gz 2024-09-17T23:41:53.511 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64005.log 2024-09-17T23:41:53.511 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.47696.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59205.log 2024-09-17T23:41:53.512 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.64005.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.40210.log.gz 2024-09-17T23:41:53.513 INFO:teuthology.orchestra.run.smithi038.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.47108.log.gz 2024-09-17T23:41:53.513 INFO:teuthology.orchestra.run.smithi038.stderr: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.64005.log.gz 2024-09-17T23:41:53.513 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52411.log 2024-09-17T23:41:53.514 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.59205.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.59205.log.gz 2024-09-17T23:41:53.515 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57954.log 2024-09-17T23:41:53.515 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.52411.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.52411.log.gz 2024-09-17T23:41:53.516 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49473.log 2024-09-17T23:41:53.517 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.57954.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.57954.log.gz 2024-09-17T23:41:53.517 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61954.log 2024-09-17T23:41:53.526 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.49473.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52584.log 2024-09-17T23:41:53.527 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.61954.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.61954.log.gz 2024-09-17T23:41:53.527 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44932.log 2024-09-17T23:41:53.528 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.52584.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.52584.log.gz 2024-09-17T23:41:53.530 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65965.log 2024-09-17T23:41:53.532 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.44932.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.44932.log.gz 2024-09-17T23:41:53.532 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45768.log 2024-09-17T23:41:53.534 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.65965.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.65965.log.gz 2024-09-17T23:41:53.534 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42218.log 2024-09-17T23:41:53.535 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.45768.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45768.log.gz 2024-09-17T23:41:53.535 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51977.log 2024-09-17T23:41:53.536 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.42218.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59949.log 2024-09-17T23:41:53.537 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.51977.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.51977.log.gz 2024-09-17T23:41:53.544 INFO:teuthology.orchestra.run.smithi038.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.41995.log.gz 2024-09-17T23:41:53.544 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51853.log 2024-09-17T23:41:53.545 INFO:teuthology.orchestra.run.smithi038.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.41488.log.gz 2024-09-17T23:41:53.548 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.59949.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.45586.log.gz 2024-09-17T23:41:53.548 INFO:teuthology.orchestra.run.smithi038.stderr: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.59949.log.gz 2024-09-17T23:41:53.548 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51109.log 2024-09-17T23:41:53.551 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.51853.log: 92.1% -- replaced with /var/log/ceph/cluster2-client.mirror.51853.log.gz 2024-09-17T23:41:53.551 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59519.log 2024-09-17T23:41:53.553 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.51109.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.51109.log.gz 2024-09-17T23:41:53.553 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54668.log 2024-09-17T23:41:53.553 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.59519.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64150.log 2024-09-17T23:41:53.553 INFO:teuthology.orchestra.run.smithi038.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.45827.log.gz 2024-09-17T23:41:53.553 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.54668.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67180.log 2024-09-17T23:41:53.555 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.64150.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.59519.log.gz 2024-09-17T23:41:53.555 INFO:teuthology.orchestra.run.smithi038.stderr: 89.4% 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.54668.log.gz 2024-09-17T23:41:53.556 INFO:teuthology.orchestra.run.smithi038.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.64150.log.gz 2024-09-17T23:41:53.556 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40701.log 2024-09-17T23:41:53.556 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.67180.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62119.log 2024-09-17T23:41:53.556 INFO:teuthology.orchestra.run.smithi038.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.67180.log.gz 2024-09-17T23:41:53.556 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.40701.log: /var/log/ceph/cluster2-client.mirror.62119.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.42518.log.gz 2024-09-17T23:41:53.557 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48355.log 2024-09-17T23:41:53.558 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48864.log 2024-09-17T23:41:53.558 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.48355.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62680.log 2024-09-17T23:41:53.558 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.48864.log: 90.2% -- replaced with /var/log/ceph/cluster2-client.mirror.62119.log.gz 2024-09-17T23:41:53.559 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43596.log 2024-09-17T23:41:53.559 INFO:teuthology.orchestra.run.smithi038.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48864.log.gz 2024-09-17T23:41:53.559 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.62680.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43478.log 2024-09-17T23:41:53.559 INFO:teuthology.orchestra.run.smithi038.stderr: 74.2% -- replaced with /var/log/ceph/cluster1-client.mirror.40701.log.gz 2024-09-17T23:41:53.561 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.43596.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.62680.log.gz 2024-09-17T23:41:53.561 INFO:teuthology.orchestra.run.smithi038.stderr: 89.7%gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41625.log 2024-09-17T23:41:53.561 INFO:teuthology.orchestra.run.smithi038.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.43596.log.gz 2024-09-17T23:41:53.561 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.43478.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61541.log 2024-09-17T23:41:53.569 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.admin.41625.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.43478.log.gz 2024-09-17T23:41:53.570 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42606.log 2024-09-17T23:41:53.570 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose --/var/log/ceph/cluster2-client.mirror.61541.log: /var/log/ceph/cluster1-client.mirror.45098.log 2024-09-17T23:41:53.570 INFO:teuthology.orchestra.run.smithi038.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.48355.log.gz 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.61541.log.gz 2024-09-17T23:41:53.570 INFO:teuthology.orchestra.run.smithi038.stderr: 2024-09-17T23:41:53.570 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.42606.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56110.log 2024-09-17T23:41:53.570 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.45098.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53843.log 2024-09-17T23:41:53.570 INFO:teuthology.orchestra.run.smithi038.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.45098.log.gz 2024-09-17T23:41:53.570 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.56110.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54953.log 2024-09-17T23:41:53.570 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.53843.log: 87.5% 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.53843.log.gz 2024-09-17T23:41:53.570 INFO:teuthology.orchestra.run.smithi038.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.56110.log.gz 2024-09-17T23:41:53.570 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63902.log 2024-09-17T23:41:53.570 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.54953.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68737.log 2024-09-17T23:41:53.570 INFO:teuthology.orchestra.run.smithi038.stderr: 87.4%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60378.log 2024-09-17T23:41:53.571 INFO:teuthology.orchestra.run.smithi038.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.54953.log.gz/var/log/ceph/cluster2-client.mirror.63902.log: 2024-09-17T23:41:53.571 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60852.log 2024-09-17T23:41:53.571 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.68737.log: /var/log/ceph/cluster1-client.mirror.60378.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.63902.log.gz 2024-09-17T23:41:53.571 INFO:teuthology.orchestra.run.smithi038.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.68737.log.gz 2024-09-17T23:41:53.571 INFO:teuthology.orchestra.run.smithi038.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.60378.log.gz 2024-09-17T23:41:53.571 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64336.log 2024-09-17T23:41:53.571 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.60852.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70282.log 2024-09-17T23:41:53.571 INFO:teuthology.orchestra.run.smithi038.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.60852.log.gz/var/log/ceph/cluster2-client.mirror.64336.log: 2024-09-17T23:41:53.571 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65740.log 2024-09-17T23:41:53.571 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.70282.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.64336.log.gz 2024-09-17T23:41:53.571 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44972.log 2024-09-17T23:41:53.577 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.65740.log: 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.70282.log.gz 2024-09-17T23:41:53.577 INFO:teuthology.orchestra.run.smithi038.stderr: 86.7%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44672.log 2024-09-17T23:41:53.577 INFO:teuthology.orchestra.run.smithi038.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.65740.log.gz 2024-09-17T23:41:53.577 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.44972.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43858.log 2024-09-17T23:41:53.577 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.44672.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.44972.log.gz 2024-09-17T23:41:53.577 INFO:teuthology.orchestra.run.smithi038.stderr: 89.7%/var/log/ceph/cluster1-client.mirror.43858.log: -- replaced with /var/log/ceph/cluster2-client.mirror.44672.log.gz 2024-09-17T23:41:53.577 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42498.log 2024-09-17T23:41:53.577 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49284.log 2024-09-17T23:41:53.577 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.42498.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.43113.log 2024-09-17T23:41:53.577 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.49284.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.41133.log 2024-09-17T23:41:53.578 INFO:teuthology.orchestra.run.smithi038.stderr: 79.4% -- replaced with /var/log/ceph/cluster1-client.mirror.49284.log.gz 2024-09-17T23:41:53.578 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56086.log 2024-09-17T23:41:53.578 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.0.43113.log: /var/log/ceph/cluster1-client.mirror.0.41133.log: /var/log/ceph/cluster1-client.mirror.56086.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.56086.log.gz 2024-09-17T23:41:53.580 INFO:teuthology.orchestra.run.smithi038.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.43858.log.gz 2024-09-17T23:41:53.580 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55910.log 2024-09-17T23:41:53.580 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61433.log 2024-09-17T23:41:53.580 INFO:teuthology.orchestra.run.smithi038.stderr:gzip/var/log/ceph/cluster1-client.mirror.55910.log: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64357.log 2024-09-17T23:41:53.581 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.61433.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.55910.log.gz 2024-09-17T23:41:53.581 INFO:teuthology.orchestra.run.smithi038.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.61433.log.gz 2024-09-17T23:41:53.582 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55580.log 2024-09-17T23:41:53.583 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.64357.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.64357.log.gz 2024-09-17T23:41:53.584 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.36756.log 2024-09-17T23:41:53.584 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61808.log 2024-09-17T23:41:53.585 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.55580.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.55580.log.gz 2024-09-17T23:41:53.585 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70574.log 2024-09-17T23:41:53.589 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.2.36756.log: /var/log/ceph/cluster2-client.mirror.61808.log: 90.2% -- replaced with /var/log/ceph/cluster2-client.mirror.61808.log.gz 2024-09-17T23:41:53.589 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64935.log 2024-09-17T23:41:53.601 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.70574.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.40581.log.gz 2024-09-17T23:41:53.601 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56999.log 2024-09-17T23:41:53.604 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.64935.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.64935.log.gz 2024-09-17T23:41:53.605 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49837.log 2024-09-17T23:41:53.607 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.56999.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.56999.log.gz 2024-09-17T23:41:53.608 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42823.log 2024-09-17T23:41:53.608 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.49837.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.49837.log.gz 2024-09-17T23:41:53.609 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50434.log 2024-09-17T23:41:53.609 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.42823.log: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.42823.log.gz 2024-09-17T23:41:53.611 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45404.log 2024-09-17T23:41:53.611 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.50434.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.50434.log.gz 2024-09-17T23:41:53.611 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49652.log 2024-09-17T23:41:53.614 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.45404.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45404.log.gz 2024-09-17T23:41:53.615 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70515.log 2024-09-17T23:41:53.618 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.49652.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.49652.log.gz 2024-09-17T23:41:53.619 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56645.log 2024-09-17T23:41:53.620 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.70515.log: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.70515.log.gz 2024-09-17T23:41:53.623 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64377.log 2024-09-17T23:41:53.624 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.56645.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.56645.log.gz 2024-09-17T23:41:53.625 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61829.log 2024-09-17T23:41:53.627 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.64377.log: 90.2% -- replaced with /var/log/ceph/cluster2-client.mirror.64377.log.gz 2024-09-17T23:41:53.628 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39657.log 2024-09-17T23:41:53.629 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.61829.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.61829.log.gz 2024-09-17T23:41:53.629 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.67554.log 2024-09-17T23:41:53.630 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.39657.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52070.log 2024-09-17T23:41:53.634 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.1.67554.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.1.67554.log.gz 2024-09-17T23:41:53.636 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48295.log 2024-09-17T23:41:53.639 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.52070.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.52070.log.gz 2024-09-17T23:41:53.639 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.49748.log 2024-09-17T23:41:53.653 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.48295.log: 74.8% -- replaced with /var/log/ceph/cluster2-client.mirror.42218.log.gz 2024-09-17T23:41:53.662 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.2.49748.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39572.log 2024-09-17T23:41:53.662 INFO:teuthology.orchestra.run.smithi038.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.39657.log.gz 2024-09-17T23:41:53.663 INFO:teuthology.orchestra.run.smithi038.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.41625.log.gz 2024-09-17T23:41:53.675 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50304.log 2024-09-17T23:41:53.682 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.39572.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40078.log 2024-09-17T23:41:53.691 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.50304.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46598.log 2024-09-17T23:41:53.694 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.40078.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50304.log.gz 2024-09-17T23:41:53.702 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62639.log 2024-09-17T23:41:53.702 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.46598.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46598.log.gz 2024-09-17T23:41:53.715 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.62639.log: 88.1%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55042.log 2024-09-17T23:41:53.716 INFO:teuthology.orchestra.run.smithi038.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.62639.log.gz 2024-09-17T23:41:53.718 INFO:teuthology.orchestra.run.smithi038.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.39572.log.gz 2024-09-17T23:41:53.721 INFO:teuthology.orchestra.run.smithi038.stderr: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.0.41133.log.gz 2024-09-17T23:41:53.732 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59294.log 2024-09-17T23:41:53.734 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.55042.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.55042.log.gz 2024-09-17T23:41:53.742 INFO:teuthology.orchestra.run.smithi038.stderr: 93.3% -- replaced with /var/log/ceph/cluster2-client.mirror.42498.log.gz 2024-09-17T23:41:53.743 INFO:teuthology.orchestra.run.smithi038.stderr: 93.7% -- replaced with /var/log/ceph/cluster1-client.mirror.2.36756.log.gz 2024-09-17T23:41:53.747 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56585.log 2024-09-17T23:41:53.749 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.59294.log: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.59294.log.gz 2024-09-17T23:41:53.772 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55339.log 2024-09-17T23:41:53.773 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.56585.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.56585.log.gz 2024-09-17T23:41:53.781 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51933.log 2024-09-17T23:41:53.782 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.55339.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.55339.log.gz 2024-09-17T23:41:53.796 INFO:teuthology.orchestra.run.smithi038.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.46921.log.gz 2024-09-17T23:41:53.804 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47974.log 2024-09-17T23:41:53.805 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.51933.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51933.log.gz 2024-09-17T23:41:53.812 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61975.log 2024-09-17T23:41:53.825 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.47974.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64253.log 2024-09-17T23:41:53.826 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.61975.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.61975.log.gz 2024-09-17T23:41:53.832 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2.log 2024-09-17T23:41:53.845 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.64253.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61725.log 2024-09-17T23:41:53.847 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2.log: 90.2% -- replaced with /var/log/ceph/cluster2-client.mirror.64253.log.gz 2024-09-17T23:41:53.852 INFO:teuthology.orchestra.run.smithi038.stderr: 87.5%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50282.log 2024-09-17T23:41:53.865 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.61725.log: -- replaced with /var/log/ceph/cluster2.log.gz 2024-09-17T23:41:53.866 INFO:teuthology.orchestra.run.smithi038.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.61725.log.gz 2024-09-17T23:41:53.876 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.50282.log: 92.0% -- replaced with /var/log/ceph/cluster1-client.mirror.50282.log.gz 2024-09-17T23:41:53.877 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45564.log 2024-09-17T23:41:53.885 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.45564.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44792.log 2024-09-17T23:41:53.887 INFO:teuthology.orchestra.run.smithi038.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.45564.log.gz 2024-09-17T23:41:53.904 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41003.log 2024-09-17T23:41:53.905 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.44792.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.44792.log.gz 2024-09-17T23:41:53.909 INFO:teuthology.orchestra.run.smithi038.stderr: 93.1% -- replaced with /var/log/ceph/cluster1-client.mirror.0.43113.log.gz 2024-09-17T23:41:53.911 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.71949.log 2024-09-17T23:41:53.911 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.41003.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43644.log 2024-09-17T23:41:53.911 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.admin.71949.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.71949.log.gz 2024-09-17T23:41:53.912 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50800.log 2024-09-17T23:41:53.912 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.43644.log: 88.5% -- replaced with /var/log/ceph/cluster1-client.mirror.43644.log.gz 2024-09-17T23:41:53.913 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52700.log 2024-09-17T23:41:53.914 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.50800.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50800.log.gz 2024-09-17T23:41:53.915 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43252.log 2024-09-17T23:41:53.915 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.52700.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.52700.log.gz 2024-09-17T23:41:53.916 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56824.log 2024-09-17T23:41:53.916 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.43252.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.43252.log.gz 2024-09-17T23:41:53.935 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67090.log 2024-09-17T23:41:53.936 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.56824.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.56824.log.gz 2024-09-17T23:41:53.936 INFO:teuthology.orchestra.run.smithi038.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49473.log.gz 2024-09-17T23:41:53.938 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44030.log 2024-09-17T23:41:53.938 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.67090.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67090.log.gz 2024-09-17T23:41:53.938 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42628.log 2024-09-17T23:41:53.941 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.44030.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.70574.log.gz 2024-09-17T23:41:53.943 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66978.log 2024-09-17T23:41:53.944 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.admin.42628.log: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.44030.log.gz 2024-09-17T23:41:53.945 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.36893.log 2024-09-17T23:41:53.946 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.66978.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.66978.log.gz 2024-09-17T23:41:53.946 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70432.log 2024-09-17T23:41:53.947 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.2.36893.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66123.log 2024-09-17T23:41:53.948 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.70432.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.70432.log.gz 2024-09-17T23:41:53.948 INFO:teuthology.orchestra.run.smithi038.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.42628.log.gz 2024-09-17T23:41:53.948 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62453.log 2024-09-17T23:41:53.949 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose --/var/log/ceph/cluster1-client.mirror.66123.log: /var/log/ceph/cluster1-client.mirror.60330.log 2024-09-17T23:41:53.950 INFO:teuthology.orchestra.run.smithi038.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.66123.log.gz 2024-09-17T23:41:53.957 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.62453.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.62453.log.gz 2024-09-17T23:41:53.958 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70096.log 2024-09-17T23:41:53.958 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.60330.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70117.log 2024-09-17T23:41:53.958 INFO:teuthology.orchestra.run.smithi038.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.60330.log.gz 2024-09-17T23:41:53.962 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.70096.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.70096.log.gz 2024-09-17T23:41:53.965 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47536.log 2024-09-17T23:41:53.967 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66055.log 2024-09-17T23:41:53.967 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.47536.log: /var/log/ceph/cluster2-client.mirror.70117.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.70117.log.gz 2024-09-17T23:41:53.967 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62556.log 2024-09-17T23:41:53.971 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.66055.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66055.log.gz 2024-09-17T23:41:53.972 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54153.log 2024-09-17T23:41:53.973 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.62556.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.62556.log.gz 2024-09-17T23:41:53.973 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49854.log 2024-09-17T23:41:53.976 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.54153.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.54153.log.gz 2024-09-17T23:41:53.976 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43436.log 2024-09-17T23:41:53.978 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.49854.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49854.log.gz 2024-09-17T23:41:53.978 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62944.log 2024-09-17T23:41:53.984 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.43436.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.43436.log.gz 2024-09-17T23:41:53.984 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48886.log 2024-09-17T23:41:53.985 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.62944.log: 91.1% -- replaced with /var/log/ceph/cluster1-client.mirror.62944.log.gz 2024-09-17T23:41:53.986 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49494.log 2024-09-17T23:41:54.000 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.48886.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39592.log 2024-09-17T23:41:54.002 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.49494.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.69156.log.gz 2024-09-17T23:41:54.004 INFO:teuthology.orchestra.run.smithi038.stderr: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49494.log.gz 2024-09-17T23:41:54.005 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46074.log 2024-09-17T23:41:54.006 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.39592.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48031.log 2024-09-17T23:41:54.007 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.46074.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46074.log.gz 2024-09-17T23:41:54.007 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65538.log 2024-09-17T23:41:54.009 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.48031.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48031.log.gz 2024-09-17T23:41:54.009 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70810.log 2024-09-17T23:41:54.013 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.65538.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.65538.log.gz 2024-09-17T23:41:54.013 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48827.log 2024-09-17T23:41:54.014 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.70810.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.70810.log.gz 2024-09-17T23:41:54.020 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63840.log 2024-09-17T23:41:54.020 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.48827.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48827.log.gz 2024-09-17T23:41:54.020 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41715.log 2024-09-17T23:41:54.021 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.63840.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.63840.log.gz 2024-09-17T23:41:54.022 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55604.log 2024-09-17T23:41:54.023 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.41715.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56014.log 2024-09-17T23:41:54.024 INFO:teuthology.orchestra.run.smithi038.stderr: 91.6% -- replaced with /var/log/ceph/cluster1-client.mirror.41715.log.gz 2024-09-17T23:41:54.024 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.55604.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.55604.log.gz 2024-09-17T23:41:54.025 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.71219.log 2024-09-17T23:41:54.025 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.56014.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.56014.log.gz 2024-09-17T23:41:54.026 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68416.log 2024-09-17T23:41:54.033 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.1.71219.log: 90.9% -- replaced with /var/log/ceph/cluster2-client.mirror.1.71219.log.gz 2024-09-17T23:41:54.033 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62845.log 2024-09-17T23:41:54.054 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.68416.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46677.log 2024-09-17T23:41:54.057 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.62845.log: 90.3% 91.7% -- replaced with /var/log/ceph/cluster1-client.mirror.2.49748.log.gz 2024-09-17T23:41:54.057 INFO:teuthology.orchestra.run.smithi038.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.62845.log.gz 2024-09-17T23:41:54.061 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45705.log 2024-09-17T23:41:54.061 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.46677.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46677.log.gz 2024-09-17T23:41:54.065 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44116.log 2024-09-17T23:41:54.066 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.45705.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.45705.log.gz 2024-09-17T23:41:54.067 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54196.log 2024-09-17T23:41:54.071 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.44116.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.2.36893.log.gz 2024-09-17T23:41:54.073 INFO:teuthology.orchestra.run.smithi038.stderr: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.44116.log.gz 2024-09-17T23:41:54.073 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42429.log 2024-09-17T23:41:54.079 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.54196.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.54196.log.gz 2024-09-17T23:41:54.079 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53936.log 2024-09-17T23:41:54.079 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.42429.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59157.log 2024-09-17T23:41:54.082 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.53936.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.53936.log.gz 2024-09-17T23:41:54.082 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48520.log 2024-09-17T23:41:54.083 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.59157.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.59157.log.gz 2024-09-17T23:41:54.083 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47756.log 2024-09-17T23:41:54.084 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.48520.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48520.log.gz 2024-09-17T23:41:54.084 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44159.log 2024-09-17T23:41:54.085 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.47756.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47756.log.gz 2024-09-17T23:41:54.085 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52242.log 2024-09-17T23:41:54.091 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.44159.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.44159.log.gz 2024-09-17T23:41:54.092 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48236.log 2024-09-17T23:41:54.095 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.52242.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.52242.log.gz 2024-09-17T23:41:54.095 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68251.log 2024-09-17T23:41:54.100 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.48236.log: 80.2% -- replaced with /var/log/ceph/cluster2-client.mirror.48236.log.gz 2024-09-17T23:41:54.101 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60282.log 2024-09-17T23:41:54.101 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.68251.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.68251.log.gz 2024-09-17T23:41:54.102 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43901.log 2024-09-17T23:41:54.102 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.60282.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.60282.log.gz 2024-09-17T23:41:54.103 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39985.log 2024-09-17T23:41:54.104 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.43901.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.45907.log.gz 2024-09-17T23:41:54.104 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49955.log 2024-09-17T23:41:54.107 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.39985.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53124.log 2024-09-17T23:41:54.108 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.49955.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.39985.log.gz 2024-09-17T23:41:54.108 INFO:teuthology.orchestra.run.smithi038.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49955.log.gz 2024-09-17T23:41:54.110 INFO:teuthology.orchestra.run.smithi038.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.43901.log.gz 2024-09-17T23:41:54.110 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42155.log 2024-09-17T23:41:54.112 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.53124.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.53124.log.gz 2024-09-17T23:41:54.112 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70075.log 2024-09-17T23:41:54.113 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.admin.42155.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69500.log 2024-09-17T23:41:54.113 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.70075.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.70075.log.gz 2024-09-17T23:41:54.114 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61658.log 2024-09-17T23:41:54.115 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.69500.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.42155.log.gz 2024-09-17T23:41:54.125 INFO:teuthology.orchestra.run.smithi038.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.47696.log.gz 2024-09-17T23:41:54.127 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55673.log 2024-09-17T23:41:54.129 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.61658.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.61658.log.gz 2024-09-17T23:41:54.140 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39708.log 2024-09-17T23:41:54.142 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.55673.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.55673.log.gz 2024-09-17T23:41:54.153 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41974.log 2024-09-17T23:41:54.159 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.admin.39708.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.39708.log.gz 2024-09-17T23:41:54.166 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63158.log 2024-09-17T23:41:54.169 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.41974.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71361.log 2024-09-17T23:41:54.170 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.63158.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.63158.log.gz 2024-09-17T23:41:54.172 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45305.log 2024-09-17T23:41:54.172 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.71361.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.71361.log.gz 2024-09-17T23:41:54.172 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64129.log 2024-09-17T23:41:54.172 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.45305.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45305.log.gz 2024-09-17T23:41:54.174 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62267.log 2024-09-17T23:41:54.174 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.64129.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58041.log 2024-09-17T23:41:54.175 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.62267.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.62267.log.gz 2024-09-17T23:41:54.176 INFO:teuthology.orchestra.run.smithi038.stderr: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.64129.log.gz 2024-09-17T23:41:54.176 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43300.log 2024-09-17T23:41:54.177 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.58041.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.58041.log.gz 2024-09-17T23:41:54.178 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70261.log 2024-09-17T23:41:54.178 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.43300.log: 89.3% -- replaced with /var/log/ceph/cluster1-client.mirror.43300.log.gz 2024-09-17T23:41:54.180 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39551.log 2024-09-17T23:41:54.180 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.70261.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.70261.log.gz 2024-09-17T23:41:54.180 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64791.log 2024-09-17T23:41:54.181 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.39551.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48075.log 2024-09-17T23:41:54.182 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.64791.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.64791.log.gz 2024-09-17T23:41:54.182 INFO:teuthology.orchestra.run.smithi038.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.39551.log.gz 2024-09-17T23:41:54.182 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68949.log 2024-09-17T23:41:54.182 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.48075.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48075.log.gz 2024-09-17T23:41:54.182 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63385.log 2024-09-17T23:41:54.183 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.68949.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49610.log 2024-09-17T23:41:54.183 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.63385.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.68949.log.gz 2024-09-17T23:41:54.183 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42381.log 2024-09-17T23:41:54.184 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.49610.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.49610.log.gz 2024-09-17T23:41:54.189 INFO:teuthology.orchestra.run.smithi038.stderr: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.63385.log.gz 2024-09-17T23:41:54.189 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46437.log 2024-09-17T23:41:54.189 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.42381.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45325.log 2024-09-17T23:41:54.189 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.46437.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.46437.log.gz 2024-09-17T23:41:54.191 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59607.log 2024-09-17T23:41:54.191 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.45325.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.45325.log.gz 2024-09-17T23:41:54.191 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52307.log 2024-09-17T23:41:54.191 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.59607.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.59607.log.gz 2024-09-17T23:41:54.192 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65515.log 2024-09-17T23:41:54.194 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.52307.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.52307.log.gz 2024-09-17T23:41:54.195 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39087.log 2024-09-17T23:41:54.195 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.65515.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.65515.log.gz 2024-09-17T23:41:54.196 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53478.log 2024-09-17T23:41:54.199 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.39087.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.39087.log.gz 2024-09-17T23:41:54.200 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53457.log 2024-09-17T23:41:54.200 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.53478.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47371.log 2024-09-17T23:41:54.201 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.53457.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.53457.log.gz 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.47974.log.gz 2024-09-17T23:41:54.202 INFO:teuthology.orchestra.run.smithi038.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.53478.log.gz 2024-09-17T23:41:54.202 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64212.log 2024-09-17T23:41:54.205 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.47371.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40263.log 2024-09-17T23:41:54.206 INFO:teuthology.orchestra.run.smithi038.stderr: 80.0% -- replaced with /var/log/ceph/cluster2-client.mirror.47371.log.gz 2024-09-17T23:41:54.206 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.64212.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37345.log 2024-09-17T23:41:54.206 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.40263.log: 2024-09-17T23:41:54.206 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53695.log 2024-09-17T23:41:54.206 INFO:teuthology.orchestra.run.smithi038.stderr: 89.4%/var/log/ceph/cluster1-client.mirror.37345.log: -- replaced with /var/log/ceph/cluster2-client.mirror.64212.log.gz 2024-09-17T23:41:54.206 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71386.log 2024-09-17T23:41:54.206 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.53695.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45123.log 2024-09-17T23:41:54.206 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.71386.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.71386.log.gz 2024-09-17T23:41:54.206 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63344.log 2024-09-17T23:41:54.212 INFO:teuthology.orchestra.run.smithi038.stderr: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.53695.log.gz 2024-09-17T23:41:54.212 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.63344.log: /var/log/ceph/cluster2-client.mirror.45123.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.45123.log.gz 2024-09-17T23:41:54.212 INFO:teuthology.orchestra.run.smithi038.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.63344.log.gz 2024-09-17T23:41:54.212 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50362.log 2024-09-17T23:41:54.212 INFO:teuthology.orchestra.run.smithi038.stderr: 54.2% -- replaced with /var/log/ceph/cluster1-client.mirror.37345.log.gz 2024-09-17T23:41:54.212 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67248.log 2024-09-17T23:41:54.212 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose --/var/log/ceph/cluster2-client.mirror.50362.log: /var/log/ceph/cluster2-client.mirror.49553.log 2024-09-17T23:41:54.213 INFO:teuthology.orchestra.run.smithi038.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.50362.log.gz 2024-09-17T23:41:54.213 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.67248.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56173.log 2024-09-17T23:41:54.213 INFO:teuthology.orchestra.run.smithi038.stderr: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.67248.log.gz/var/log/ceph/cluster2-client.mirror.49553.log: 2024-09-17T23:41:54.213 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56761.log 2024-09-17T23:41:54.213 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.56173.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55693.log 2024-09-17T23:41:54.213 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.56761.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.56761.log.gz 2024-09-17T23:41:54.213 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65268.log 2024-09-17T23:41:54.218 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.55693.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.55693.log.gz 2024-09-17T23:41:54.218 INFO:teuthology.orchestra.run.smithi038.stderr: 89.5%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.42218.log 2024-09-17T23:41:54.218 INFO:teuthology.orchestra.run.smithi038.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.56173.log.gz 2024-09-17T23:41:54.218 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54784.log 2024-09-17T23:41:54.218 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.65268.log: /var/log/ceph/cluster1-client.mirror.0.42218.log: 86.7%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64315.log 2024-09-17T23:41:54.218 INFO:teuthology.orchestra.run.smithi038.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.65268.log.gz 2024-09-17T23:41:54.218 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.54784.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37552.log 2024-09-17T23:41:54.218 INFO:teuthology.orchestra.run.smithi038.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.54784.log.gz 2024-09-17T23:41:54.219 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49246.log 2024-09-17T23:41:54.219 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.64315.log: /var/log/ceph/cluster2-client.admin.37552.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52980.log 2024-09-17T23:41:54.219 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.49246.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49246.log.gz 2024-09-17T23:41:54.223 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40611.log 2024-09-17T23:41:54.223 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.52980.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.64315.log.gz 2024-09-17T23:41:54.223 INFO:teuthology.orchestra.run.smithi038.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.52980.log.gz 2024-09-17T23:41:54.223 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70975.log 2024-09-17T23:41:54.223 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69076.log 2024-09-17T23:41:54.223 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.70975.log: /var/log/ceph/cluster1-client.admin.40611.log: gzip 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.70975.log.gz 2024-09-17T23:41:54.223 INFO:teuthology.orchestra.run.smithi038.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69358.log 2024-09-17T23:41:54.223 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56038.log 2024-09-17T23:41:54.223 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.69358.log: /var/log/ceph/cluster2-client.mirror.69076.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.69358.log.gz 2024-09-17T23:41:54.223 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61453.log 2024-09-17T23:41:54.224 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.56038.log: 91.1% -- replaced with /var/log/ceph/cluster2-client.mirror.69076.log.gz 2024-09-17T23:41:54.226 INFO:teuthology.orchestra.run.smithi038.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.56038.log.gz 2024-09-17T23:41:54.226 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40033.log 2024-09-17T23:41:54.228 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.61453.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.61453.log.gz 2024-09-17T23:41:54.228 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42015.log 2024-09-17T23:41:54.229 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.40033.log: gzip 66.7% -- replaced with /var/log/ceph/cluster2-client.admin.37552.log.gz 2024-09-17T23:41:54.229 INFO:teuthology.orchestra.run.smithi038.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.admin.37144.log 2024-09-17T23:41:54.231 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.42015.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50820.log 2024-09-17T23:41:54.232 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.admin.37144.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37144.log.gz 2024-09-17T23:41:54.232 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68970.log 2024-09-17T23:41:54.233 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.50820.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.50820.log.gz 2024-09-17T23:41:54.234 INFO:teuthology.orchestra.run.smithi038.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.40611.log.gz 2024-09-17T23:41:54.236 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59991.log 2024-09-17T23:41:54.239 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.68970.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.40033.log.gz 2024-09-17T23:41:54.239 INFO:teuthology.orchestra.run.smithi038.stderr: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.68970.log.gz 2024-09-17T23:41:54.239 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54060.log 2024-09-17T23:41:54.240 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.59991.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.59991.log.gz 2024-09-17T23:41:54.241 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37260.log 2024-09-17T23:41:54.242 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.54060.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.54060.log.gz 2024-09-17T23:41:54.242 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66348.log 2024-09-17T23:41:54.243 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.37260.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68336.log 2024-09-17T23:41:54.243 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster1-client.mirror.66348.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66348.log.gz 2024-09-17T23:41:54.246 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cluster2-client.mirror.68336.log: 91.1% -- replaced with /var/log/ceph/cluster2-client.mirror.68336.log.gz 2024-09-17T23:41:54.254 INFO:teuthology.orchestra.run.smithi038.stderr: 2.3% -- replaced with /var/log/ceph/cluster1-client.mirror.37260.log.gz 2024-09-17T23:41:54.312 INFO:teuthology.orchestra.run.smithi038.stderr: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.41003.log.gz 2024-09-17T23:41:54.330 INFO:teuthology.orchestra.run.smithi038.stderr: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.0.42218.log.gz 2024-09-17T23:41:54.337 INFO:teuthology.orchestra.run.smithi038.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.47536.log.gz 2024-09-17T23:41:54.422 INFO:teuthology.orchestra.run.smithi038.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.42606.log.gz 2024-09-17T23:41:54.443 INFO:teuthology.orchestra.run.smithi038.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48295.log.gz 2024-09-17T23:41:54.537 INFO:teuthology.orchestra.run.smithi038.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49553.log.gz 2024-09-17T23:41:54.551 INFO:teuthology.orchestra.run.smithi038.stderr: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.42015.log.gz 2024-09-17T23:41:54.608 INFO:teuthology.orchestra.run.smithi038.stderr: 91.2% -- replaced with /var/log/ceph/cluster2-client.mirror.40263.log.gz 2024-09-17T23:41:54.617 INFO:teuthology.orchestra.run.smithi038.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48886.log.gz 2024-09-17T23:41:54.639 INFO:teuthology.orchestra.run.smithi038.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.68416.log.gz 2024-09-17T23:41:54.695 INFO:teuthology.orchestra.run.smithi038.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.41974.log.gz 2024-09-17T23:41:54.705 INFO:teuthology.orchestra.run.smithi038.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.69500.log.gz 2024-09-17T23:41:54.797 INFO:teuthology.orchestra.run.smithi038.stderr: 88.9% -- replaced with /var/log/ceph/cluster1-client.mirror.42429.log.gz 2024-09-17T23:41:54.824 INFO:teuthology.orchestra.run.smithi038.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.39592.log.gz 2024-09-17T23:41:54.838 INFO:teuthology.orchestra.run.smithi038.stderr: 88.9% -- replaced with /var/log/ceph/cluster1-client.mirror.42381.log.gz 2024-09-17T23:41:55.208 INFO:teuthology.orchestra.run.smithi038.stderr: 93.3% -- replaced with /var/log/ceph/cluster2-client.mirror.40078.log.gz 2024-09-17T23:41:56.155 INFO:teuthology.orchestra.run.smithi038.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.42104.log.gz 2024-09-17T23:41:57.777 INFO:teuthology.orchestra.run.smithi038.stderr: 90.0% -- replaced with /var/log/ceph/cluster2-mon.a.log.gz 2024-09-17T23:41:58.514 INFO:teuthology.orchestra.run.smithi006.stderr: 94.3% -- replaced with /var/log/ceph/cluster1-osd.1.log.gz 2024-09-17T23:41:58.515 INFO:teuthology.orchestra.run.smithi038.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.41595.log.gz 2024-09-17T23:41:58.663 INFO:teuthology.orchestra.run.smithi006.stderr: 94.4% -- replaced with /var/log/ceph/cluster1-osd.0.log.gz 2024-09-17T23:41:58.758 INFO:teuthology.orchestra.run.smithi006.stderr: 94.4% -- replaced with /var/log/ceph/cluster1-osd.2.log.gz 2024-09-17T23:41:58.759 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-17T23:41:58.760 INFO:teuthology.orchestra.run.smithi006.stderr:real 0m9.216s 2024-09-17T23:41:58.760 INFO:teuthology.orchestra.run.smithi006.stderr:user 0m27.926s 2024-09-17T23:41:58.760 INFO:teuthology.orchestra.run.smithi006.stderr:sys 0m0.968s 2024-09-17T23:42:00.113 INFO:teuthology.orchestra.run.smithi038.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.38307.log.gz 2024-09-17T23:42:02.260 INFO:teuthology.orchestra.run.smithi038.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.40185.log.gz 2024-09-17T23:42:02.291 INFO:teuthology.orchestra.run.smithi038.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.41090.log.gz 2024-09-17T23:42:04.616 INFO:teuthology.orchestra.run.smithi038.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.39679.log.gz 2024-09-17T23:42:08.661 INFO:teuthology.orchestra.run.smithi038.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.39268.log.gz 2024-09-17T23:43:54.578 INFO:teuthology.orchestra.run.smithi038.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-osd.1.log.gz 2024-09-17T23:44:12.702 INFO:teuthology.orchestra.run.smithi038.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-osd.2.log.gz 2024-09-17T23:44:28.948 INFO:teuthology.orchestra.run.smithi038.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-osd.0.log.gz 2024-09-17T23:44:28.950 INFO:teuthology.orchestra.run.smithi038.stderr: 2024-09-17T23:44:28.950 INFO:teuthology.orchestra.run.smithi038.stderr:real 2m39.367s 2024-09-17T23:44:28.950 INFO:teuthology.orchestra.run.smithi038.stderr:user 4m17.598s 2024-09-17T23:44:28.950 INFO:teuthology.orchestra.run.smithi038.stderr:sys 0m10.466s 2024-09-17T23:44:28.950 INFO:tasks.ceph:Archiving logs... 2024-09-17T23:44:28.951 DEBUG:teuthology.misc:Transferring archived files from smithi006:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-17_21:16:02-rbd-squid-distro-default-smithi/7909572/remote/smithi006/log 2024-09-17T23:44:28.952 DEBUG:teuthology.orchestra.run.smithi006:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-17T23:44:32.622 DEBUG:teuthology.misc:Transferring archived files from smithi038:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-17_21:16:02-rbd-squid-distro-default-smithi/7909572/remote/smithi038/log 2024-09-17T23:44:32.623 DEBUG:teuthology.orchestra.run.smithi038:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-17T23:45:09.934 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-17T23:45:09.947 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-17T23:45:09.948 DEBUG:teuthology.orchestra.run.smithi006:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-17T23:45:09.951 DEBUG:teuthology.orchestra.run.smithi038:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-17T23:45:10.047 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-17T23:45:10.058 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-17T23:45:10.058 DEBUG:teuthology.orchestra.run.smithi006:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-17T23:45:10.061 DEBUG:teuthology.orchestra.run.smithi038:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-17T23:45:10.077 INFO:teuthology.orchestra.run.smithi006.stderr:bash: line 1: ntpq: command not found 2024-09-17T23:45:10.079 INFO:teuthology.orchestra.run.smithi038.stderr:bash: line 1: ntpq: command not found 2024-09-17T23:45:10.081 INFO:teuthology.orchestra.run.smithi006.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-17T23:45:10.081 INFO:teuthology.orchestra.run.smithi006.stdout:=============================================================================== 2024-09-17T23:45:10.081 INFO:teuthology.orchestra.run.smithi006.stdout:^* hv01.front.sepia.ceph.com 3 6 377 27 -118us[ -114us] +/- 52ms 2024-09-17T23:45:10.082 INFO:teuthology.orchestra.run.smithi006.stdout:^+ hv02.front.sepia.ceph.com 3 7 377 27 -792us[ -787us] +/- 60ms 2024-09-17T23:45:10.082 INFO:teuthology.orchestra.run.smithi006.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 26 +1278us[+1278us] +/- 90ms 2024-09-17T23:45:10.082 INFO:teuthology.orchestra.run.smithi006.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T23:45:10.113 INFO:teuthology.orchestra.run.smithi038.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-17T23:45:10.113 INFO:teuthology.orchestra.run.smithi038.stdout:=============================================================================== 2024-09-17T23:45:10.113 INFO:teuthology.orchestra.run.smithi038.stdout:^* hv01.front.sepia.ceph.com 3 6 377 25 -49us[ -56us] +/- 52ms 2024-09-17T23:45:10.113 INFO:teuthology.orchestra.run.smithi038.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 27 -754us[ -762us] +/- 60ms 2024-09-17T23:45:10.113 INFO:teuthology.orchestra.run.smithi038.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 26 +1346us[+1338us] +/- 90ms 2024-09-17T23:45:10.113 INFO:teuthology.orchestra.run.smithi038.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T23:45:10.114 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-17T23:45:10.126 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-17T23:45:10.126 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-17T23:45:10.137 DEBUG:teuthology.orchestra.run.smithi006:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-17T23:45:10.209 DEBUG:teuthology.orchestra.run.smithi038:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-17T23:45:10.311 DEBUG:teuthology.orchestra.run.smithi006:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-17T23:45:10.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T23:45:10.344 DEBUG:teuthology.orchestra.run.smithi038:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-17T23:45:10.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T23:45:10.375 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-17T23:45:10.386 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-17T23:45:10.397 INFO:teuthology.task.internal:Duration was 2852.082978 seconds 2024-09-17T23:45:10.397 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-17T23:45:10.409 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-17T23:45:10.409 DEBUG:teuthology.orchestra.run.smithi006:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-17T23:45:10.412 DEBUG:teuthology.orchestra.run.smithi038:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-17T23:45:10.455 INFO:teuthology.orchestra.run.smithi006.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-17T23:45:10.493 INFO:teuthology.orchestra.run.smithi038.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-17T23:45:10.659 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-17T23:45:10.659 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi006.front.sepia.ceph.com 2024-09-17T23:45:10.660 DEBUG:teuthology.orchestra.run.smithi006:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-17T23:45:10.683 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi038.front.sepia.ceph.com 2024-09-17T23:45:10.684 DEBUG:teuthology.orchestra.run.smithi038:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-17T23:45:10.707 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-17T23:45:10.707 DEBUG:teuthology.orchestra.run.smithi006:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-17T23:45:10.727 DEBUG:teuthology.orchestra.run.smithi038:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-17T23:45:10.777 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-17T23:45:10.777 DEBUG:teuthology.orchestra.run.smithi006:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-17T23:45:10.796 DEBUG:teuthology.orchestra.run.smithi038:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-17T23:45:10.933 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-17T23:45:10.946 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-17T23:45:10.946 DEBUG:teuthology.orchestra.run.smithi006:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-17T23:45:10.973 DEBUG:teuthology.orchestra.run.smithi038:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-17T23:45:11.011 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-17T23:45:11.023 DEBUG:teuthology.orchestra.run.smithi006:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-17T23:45:11.026 DEBUG:teuthology.orchestra.run.smithi038:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-17T23:45:11.047 INFO:teuthology.orchestra.run.smithi006.stdout:kernel.core_pattern = core 2024-09-17T23:45:11.086 INFO:teuthology.orchestra.run.smithi038.stdout:kernel.core_pattern = core 2024-09-17T23:45:11.101 DEBUG:teuthology.orchestra.run.smithi006:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-17T23:45:11.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T23:45:11.121 DEBUG:teuthology.orchestra.run.smithi038:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-17T23:45:11.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T23:45:11.159 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-17T23:45:11.172 INFO:teuthology.task.internal:Transferring archived files... 2024-09-17T23:45:11.173 DEBUG:teuthology.misc:Transferring archived files from smithi006:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-17_21:16:02-rbd-squid-distro-default-smithi/7909572/remote/smithi006 2024-09-17T23:45:11.173 DEBUG:teuthology.orchestra.run.smithi006:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-17T23:45:11.208 DEBUG:teuthology.misc:Transferring archived files from smithi038:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-17_21:16:02-rbd-squid-distro-default-smithi/7909572/remote/smithi038 2024-09-17T23:45:11.209 DEBUG:teuthology.orchestra.run.smithi038:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-17T23:45:11.245 INFO:teuthology.task.internal:Removing archive directory... 2024-09-17T23:45:11.245 DEBUG:teuthology.orchestra.run.smithi006:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-17T23:45:11.248 DEBUG:teuthology.orchestra.run.smithi038:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-17T23:45:11.297 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-17T23:45:11.319 INFO:teuthology.task.internal:Not uploading archives. 2024-09-17T23:45:11.319 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-17T23:45:11.337 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-17T23:45:11.337 DEBUG:teuthology.orchestra.run.smithi006:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-17T23:45:11.340 DEBUG:teuthology.orchestra.run.smithi038:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-17T23:45:11.353 INFO:teuthology.orchestra.run.smithi006.stdout: 265607 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 17 23:45 /home/ubuntu/cephtest 2024-09-17T23:45:11.356 INFO:teuthology.orchestra.run.smithi038.stdout: 266055 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 17 23:45 /home/ubuntu/cephtest 2024-09-17T23:45:11.357 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-17T23:45:11.369 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-17T23:45:11.397 INFO:teuthology.run:Summary data: description: rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-zstd policy/none rbd-mirror/four-per-cluster supported-random-distro$/{centos_latest} workloads/rbd-mirror-snapshot-stress-workunit-fast-diff} duration: 2852.0829775333405 failure_reason: '"2024-09-17T23:07:26.087572+0000 mon.a (mon.0) 208 : cluster [WRN] Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY)" in cluster log' flavor: default owner: scheduled_teuthology@teuthology success: false 2024-09-17T23:45:11.397 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-17T23:45:11.503 INFO:teuthology.run:FAIL